百度 末节前半段，两队的进攻效率都不高，勇士稍稍占优继续维持领先。

When developing and maintaining OpenResty applications, you’ve likely encountered these frustrating scenarios: errors in Lua code that are difficult to trace; performance bottlenecks hiding within complex call stacks; intermittent issues in production environments that are nearly impossible to reproduce in development; and when applications crash, they leave only fragments of error messages without the complete context. Traditional debugging tools often leave us feeling helpless—once you miss the critical moment, you have to start over, which is both inefficient and risks missing the root cause of the problem.

With the powerful combination of UDB (Undo Debugger) and OpenResty XRay, we finally have the ability to freely navigate through a program’s execution history, like having a “time machine” for the debugging world. Whether you’re tracking memory allocation patterns, analyzing execution paths for specific requests, or pinpointing code segments that cause crashes, this revolutionary debugging approach provides a panoramic view that traditional tools simply cannot offer. Let’s explore how this breakthrough technology transforms our problem-solving approach through a real-world OpenResty application case study.

How OpenResty XRay Enhances Efficiency with UDB

How UDB Unveils the Mystery of Lua Code in OpenResty Applications

As a time-travel debugging tool launched by Undo, UDB provides development teams with unprecedented visualization capabilities:

• Exploration in the Time Dimension: Say goodbye to the tedious “reproduce-fix-verify” cycle. You can freely navigate through the program execution trajectory, capturing every critical moment.
• Insight into the Root Cause: Through intelligent breakpoints and monitoring mechanisms, UDB directly targets the triggering conditions of abnormal behaviors, transforming vague error reports into clear problem profiles.
• Transparency of Runtime States: Real-time analysis of Lua variables and memory structures reveals the internal workings of OpenResty applications, quickly identifying anomalous patterns.
• Perfect Compatibility with the Lua Ecosystem: UDB seamlessly integrates with the OpenResty environment, providing Lua developers with a professional-grade debugging experience and supporting multi-dimensional analysis of complex applications.

Powerful Alliance with OpenResty XRay, Creating a Comprehensive Debugging System

OpenResty XRay automatically analyzes the runtime behavior of Lua code in OpenResty applications, precisely locating various technical pain points. When combined with UDB, it brings you:

• Panoramic Problem Diagnosis: A three-dimensional view from request handling to underlying execution
• Lua Code Performance Profiling: Intuitive display of hotspot functions and resource consumption
• Zero-Interference Analysis Experience: Gain deep insights while maintaining the application’s original behavior
• Practice-Oriented Optimization Guidelines: Targeted improvement strategies based on real-world workloads

Through the synergy between UDB and OpenResty XRay, development teams can quickly break through technical bottlenecks, transforming complex troubleshooting into clear solution paths. This combination not only accelerates problem-solving processes but also fundamentally enhances the quality and stability of OpenResty applications, providing solid technical support for continuous business innovation.

Hands-On: Analyzing OpenResty Lua Call Stacks with UDB and OpenResty XRay

Let’s dive into a practical case study demonstrating how to precisely analyze call stacks in OpenResty Lua applications using UDB:

Step 1: Recording Application Execution Traces and Replaying Recorded Samples

First, we’ll use UDB’s Live Record tool to capture the execution process of an OpenResty application:

1. Use the Live Record tool to record a sample of a running OpenResty application. 1.1 Select “Live Recording” in the OpenResty XRay console. 1.2 Choose the target application and process, then click “Start Recording”. 1.3 Click the “Generate Recording File” icon to create a recording file. 1.4 After generation is complete, click “Stop Recording” to end the recording process. 1.5 Download the recording file for analysis.

2. Compile the relevant tools on OpenResty XRay, and download the compiled tools to your local environment.

3. Load the recorded sample using the UDB tool and set up the debugging environment:

udb --sessions=no -ex "set pagination off" -ex "set python print-stack full" openresty.rec

Step 2: Analyzing Recorded Samples

This article uses Lua memory allocation analysis as an example to demonstrate how to view OpenResty Lua execution call stacks in udb. We’ll set a breakpoint at the lua_alloc_realloc function.

0% 51> b lj_alloc_realloc
Breakpoint 1 at 0x55560961bca0: file lj_alloc.c, line 1520.
0% 51> c
Continuing.
Thread 1 "nginx" hit Breakpoint 1, lj_alloc_realloc (msp=0x7f5657a9d010, ptr=ptr@entry=0x7f549be49108, nsize=40) at lj_alloc.c:1520
1520    lj_alloc.c: No such file or directory.

Step 3: Analyzing Low-Level C Call Stacks

4. Use the bt command to view the current C-level call stack:

0% 853> bt
#0  lj_alloc_realloc (msp=0x7f5657a9d010, ptr=ptr@entry=0x7f549be49108, nsize=40) at lj_alloc.c:1520
#1  0x000055560961c053 in lj_alloc_f (msp=<optimized out>, ptr=0x7f549be49108, osize=<optimized out>, nsize=<optimized out>) at lj_alloc.c:1582
#2  0x0000555609625270 in lj_mem_realloc (L=0x7f54b4bd1628, p=p@entry=0x7f549be49108, osz=osz@entry=24, nsz=40) at lj_gc.c:883
#3  0x0000555609626e3d in lj_tab_resize (L=0x7f54b4bd1628, t=0x7f54b5065cf0, asize=5, hbits=0) at lj_tab.c:256
#4  0x0000555609626986 in rehashtab (ek=<optimized out>, t=<optimized out>, L=<optimized out>) at lj_tab.c:375
#5  lj_tab_newkey (L=0x7f54b4bd1628, t=0x7f54b5065cf0, key=0x7f549ea60380) at lj_tab.c:453
#6  0x00005556469832df in ?? ()
#7  0x00007ffe8a54b9f0 in ?? ()
#8  0x000055560957e8dd in ngx_lua_conf_ffi_get (keys=<optimized out>, keys_len=0x7f5656ab5dd0, values=0x7f5657a80c78 "x\003\250WV\177", values_len=0x7f5656ab5e70, 
    db=<optimized out>) at ../lua-resty-config-1.0.6/src/ngx_lua_config_module.c:389
#9  0x0000001200000000 in ?? ()
#10 0x00007f549bc917a8 in ?? ()
#11 0x00007f549bc917b6 in ?? ()
#12 0x00007f549cf27568 in ?? ()
#13 0x00007f54b547d508 in ?? ()
#14 0x00007f54b6825e30 in ?? ()
#15 0x00007f549eb0d1a0 in ?? ()
#16 0x00007f549cf275a8 in ?? ()

Looking at the C call stack, we can see that the system’s low-level lj_alloc_realloc function call is triggered, but this only shows information at the C language level and doesn’t provide the complete call path of the Lua business code.

Step 4: Analyzing the Complete Call Stack of Lua Code

5. Load the Lua call stack analysis tool provided by OpenResty XRay:

4% 7,955> source lj-lua-on-cpu.y.py

6. Use the lj_lua_on_cpu command to obtain the complete Lua call stack:

0% 853> lj_lua_on_cpu 
C:lj_alloc_realloc
trace#6786:http-runtime-20181201.lua:4028
@1:890
helper_5
@1:1538
[builtin#xpcall]
xpcall
@1:1424
run_access_phase
@edge.lua:187
access
@access_by_lua(nginx.conf:766):2

Through this complete Lua call stack, we can clearly see that the business code is running in the access_by_lua phase.

Currently, the code being executed is trace 6786, which is just-in-time compiled code. This explains why the C call stack obtained from the previous bt command didn’t properly expand after the ngx_lua_conf_ffi_get function.

Advantages of Time-Travel Debugging

Time-travel debugging is one of UDB’s most powerful features. With this capability, we can freely move forward or backward through the recorded execution trace, precisely pinpoint different memory allocation operation timestamps, and comprehensively analyze the complete context and call stack for each allocation operation.

To verify this, we can continue executing the program to capture the next lj_alloc_realloc call:

0% 51> c
Continuing.
Thread 1 "nginx" hit Breakpoint 1, lj_alloc_realloc (msp=0x7f5657a9d010, ptr=ptr@entry=0x7f549be49108, nsize=40) at lj_alloc.c:1520
1520    lj_alloc.c: No such file or directory.
1% 849,555> lj_lua_on_cpu 
C:lj_alloc_realloc
trace#177:common-runtime-20181201.lua:393
@common-runtime-20181201.lua:383
bit32_to_strtbl
@common-runtime-20181201.lua:435
inet_to_bitstr
@1:1459
[builtin#xpcall]
xpcall
@1:1424
run_rewrite_phase
@edge.lua:187
access
@access_by_lua(nginx.conf:766):2

By comparing two captured call stacks, we can instantly identify the key differences:

1. The first memory allocation comes from trace 6786
2. The second memory allocation comes from trace 177

This precision targeting capability isn’t just applicable to memory allocation analysis—it’s a powerful tool for solving complex problems. Whether you’re analyzing the Lua code execution path for specific URI requests or tracking down Lua code fragments that cause process crashes, this method handles these challenges with ease.

Traditional GDB coredump analysis is like a snapshot of an accident scene—you can only see the final result. UDB, on the other hand, is like high-definition video playback, allowing you to observe every detail leading up to the incident. Even when the original process is no longer running, you can navigate the complete program execution history effortlessly, examining program states and behavior details at any point in time with precision. This truly delivers a comprehensive “time travel” debugging experience.

Summary

Through the practical exploration in this article, we’ve demonstrated how UDB’s time-travel debugging capabilities, combined with OpenResty XRay analysis tools, can deeply dissect the execution process of Lua code in OpenResty applications. This powerful combination offers several significant advantages:

1. Comprehensive Execution Context: Not only can you see the underlying C call stack, but you can also precisely track the complete call path of Lua business code, including the execution of just-in-time (JIT) compiled code.

2. Time-Dimensional Debugging Capabilities: UDB allows you to freely navigate through the program’s execution history without restarting the application, enabling analysis of program states at different points in time. This is particularly valuable for troubleshooting intermittent issues and complex scenarios.

3. Deep Performance Analysis: Combined with OpenResty XRay’s analysis tools, you can precisely identify performance bottlenecks, such as memory allocation patterns and hotspot functions.

4. Post-Mortem Analysis Capabilities: Even after the original process has terminated, you can still perform comprehensive analysis through recorded samples. This far exceeds the limitations of traditional coredump analysis, which can only provide a static snapshot at the moment of a crash.

For software developers and operations experts striving for excellence, mastering this cutting-edge debugging technology is no longer optional but a necessary skill to stay ahead in the competitive tech landscape. Through the powerful combination of UDB and OpenResty XRay, you can understand application behavior with unprecedented precision, enabling you to build more stable, high-performance application systems.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^? open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^?, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.