1. LTTng and Erlang/OTP

1 LTTng和Erlang / OTP

1.1简介

Linux跟踪工具包：下一代是一个开源系统软件包，用于对Linux内核，用户应用程序和库进行相关跟踪。

欲知详情，请浏览http://lttng.org

1.2构建支持LTTng的Erlang / OTP

使用LTTNG支持配置和构建Erlang：

要使LTTNG与Erlang/OTP一起正常工作，需要安装以下软件包：

LTTng-Tools：用于控制跟踪会话的命令行接口。

LTTng-UST：用户空间跟踪库。

在Ubuntu上，可以通过aptitude*

$ sudo aptitude install lttng-tools liblttng-ust-dev

请参阅Installing LTTng有关如何在系统上安装LTTng的更多信息。

在系统上正确安装LTTNG之后，可以使用LTTNG支持构建Erlang/OTP。

$ ./configure --with-dynamic-trace=lttng
$ make

1.3 Dyntrace跟踪点

所有跟踪点都在org_erlang_dyntrace

所有Erlang类型都是LTTNG中等效的字符串。

process_spawn

pid : string :: Process ID. Ex. "<0.131.0>"

parent : string :: Process ID. Ex. "<0.131.0>"

entry : string :: Code Location. Ex. "lists:sort/1"

通过erlang:trace/3跟踪标志procs和{tracer,dyntrace,[]}示踪模块可用。

例子：

process_spawn: { cpu_id = 3 }, { pid = "<0.131.0>", parent = "<0.130.0>", entry = "erlang:apply/2" }

process_link

to : string :: Process ID or Port ID. Ex. "<0.131.0>"

from : string :: Process ID or Port ID. Ex.<0.131.0>"

type : string*"link" | "unlink"

通过erlang:trace/3跟踪标志procs和{tracer,dyntrace,[]}示踪模块可用。

例子：

process_link: { cpu_id = 3 }, { from = "<0.130.0>", to = "<0.131.0>", type = "link" }

process_exit

pid : string :: Process ID. Ex. "<0.131.0>"

reason : string :: Exit reason. Ex. "normal"

通过erlang:trace/3跟踪标志procs和{tracer,dyntrace,[]}示踪模块可用。

例子：

process_exit: { cpu_id = 3 }, { pid = "<0.130.0>", reason = "normal" }

process_register

pid : string :: Process ID. Ex. "<0.131.0>"

name : string :: Registered name. Ex. "error_logger"

type : string*"register" | "unregister"

例子：

process_register: { cpu_id = 0 }, { pid = "<0.128.0>", name = "dyntrace_lttng_SUITE" type = "register" }

process_scheduled

pid : string :: Process ID. Ex. "<0.131.0>"

entry : string :: Code Location. Ex. "lists:sort/1"

type : string*"in" | "out" | "in_exiting" | "out_exiting" | "out_exited"

通过erlang:trace/3跟踪标志running和{tracer,dyntrace,[]}示踪模块可用。

例子：

process_scheduled: { cpu_id = 0 }, { pid = "<0.136.0>", entry = "erlang:apply/2", type = "in" }

port_open

pid : string :: Process ID. Ex. "<0.131.0>"

driver : string :: Driver name. Ex. "efile"

port : string :: Port ID. Ex. "#Port<0.1031>"

通过erlang:trace/3跟踪标志ports和{tracer,dyntrace,[]}示踪模块可用。

例子：

port_open: { cpu_id = 5 }, { pid = "<0.131.0>", driver = "'/bin/sh -s unix:cmd'", port = "#Port<0.1887>" }

port_exit

port : string :: Port ID. Ex. "#Port<0.1031>"

reason : string :: Exit reason. Ex. "normal"

通过erlang:trace/3跟踪标志ports和{tracer,dyntrace,[]}示踪模块可用。

例子：

port_exit: { cpu_id = 5 }, { port = "#Port<0.1887>", reason = "normal" }

port_link

to : string :: Process ID. Ex. "<0.131.0>"

from : string :: Process ID. Ex. "<0.131.0>"

type : string*"link" | "unlink"

通过erlang:trace/3跟踪标志ports和{tracer,dyntrace,[]}示踪模块可用。

例子：

port_link: { cpu_id = 5 }, { from = "#Port<0.1887>", to = "<0.131.0>", type = "unlink" }

port_scheduled

通过erlang:trace/3跟踪标志running和{tracer,dyntrace,[]}示踪模块可用。

port : string :: Port ID. Ex. "#Port<0.1031>"

entry : string :: Callback. Ex. "open"

type : string*"in" | "out" | "in_exiting" | "out_exiting" | "out_exited"

例子：

port_scheduled: { cpu_id = 5 }, { pid = "#Port<0.1905>", entry = "close", type = "out" }

通过erlang:trace/3跟踪标志running和{tracer,dyntrace,[]}示踪模块可用。

function_call

pid : string :: Process ID. Ex. "<0.131.0>"

entry : string :: Code Location. Ex. "lists:sort/1"

depth : integer :: Stack depth. Ex. 0

通过erlang:trace/3跟踪标志call和{tracer,dyntrace,[]}示踪模块可用。

例子：

function_call: { cpu_id = 5 }, { pid = "<0.145.0>", entry = "dyntrace_lttng_SUITE:'-t_call/1-fun-1-'/0", depth = 0 }

function_return

pid : string :: Process ID. Ex. "<0.131.0>"

entry : string :: Code Location. Ex. "lists:sort/1"

depth : integer :: Stack depth. Ex. 0

可通过erlang:trace/3使用跟踪标志call或return_to和{tracer,dyntrace,[]}作为示踪模块。

例子：

function_return: { cpu_id = 5 }, { pid = "<0.145.0>", entry = "dyntrace_lttng_SUITE:waiter/0", depth = 0 }

function_exception

pid : string :: Process ID. Ex. "<0.131.0>"

entry : string :: Code Location. Ex. "lists:sort/1"

class : string :: Error reason. Ex. "error"

通过erlang:trace/3跟踪标志call和{tracer,dyntrace,[]}示踪模块可用。

例子：

function_exception: { cpu_id = 5 }, { pid = "<0.144.0>", entry = "t:call_exc/1", class = "error" }

message_send

from : string :: Process ID or Port ID. Ex. "<0.131.0>"

to : string :: Process ID or Port ID. Ex. "<0.131.0>"

message : string :: Message sent. Ex. "{<0.162.0>,ok}"

通过erlang:trace/3跟踪标志send和{tracer,dyntrace,[]}示踪模块可用。

例子：

message_send: { cpu_id = 3 }, { from = "#Port<0.1938>", to = "<0.160.0>", message = "{#Port<0.1938>,eof}" }

message_receive

to : string :: Process ID or Port ID. Ex. "<0.131.0>"

message : string :: Message received. Ex. "{<0.162.0>,ok}"

通过erlang:trace/3跟踪标志'receive'和{tracer,dyntrace,[]}示踪模块可用。

例子：

message_receive: { cpu_id = 7 }, { to = "<0.167.0>", message = "{<0.165.0>,ok}" }

gc_minor_start

pid : string :: Process ID. Ex. "<0.131.0>"

need : integer :: Heap need. Ex. 2

heap : integer::年轻的堆字大小。Ex.233

old_heap : integer::旧堆字大小。Ex.233

通过erlang:trace/3跟踪标志garbage_collection和{tracer,dyntrace,[]}示踪模块可用。

例子：

gc_minor_start: { cpu_id = 0 }, { pid = "<0.172.0>", need = 0, heap = 610, old_heap = 0 }

gc_minor_end

pid : string::进程ID。Ex."<0.131.0>"

reclaimed : integer::堆回收。Ex.2

heap : integer::较新的堆字大小。Ex.233

old_heap : integer::旧堆字大小。Ex.233

通过erlang:trace/3跟踪标志garbage_collection和{tracer,dyntrace,[]}示踪模块可用。

例子：

gc_minor_end: { cpu_id = 0 }, { pid = "<0.172.0>", reclaimed = 120, heap = 1598, old_heap = 1598 }

gc_major_start

pid : string::进程ID。Ex."<0.131.0>"

need : integer::堆需要。Ex.2

heap : integer*较新的堆字大小。Ex.233

old_heap : integer*旧堆字大小。前。233

通过erlang:trace/3跟踪标志garbage_collection和{tracer,dyntrace,[]}示踪模块可用。

例子：

gc_major_start: { cpu_id = 0 }, { pid = "<0.172.0>", need = 8, heap = 2586, old_heap = 1598 }

gc_major_end

pid : string*进程ID。Ex."<0.131.0>"

reclaimed : integer*堆回收。Ex.2

heap : integer*较新的堆字大小。Ex.233

old_heap : integer*旧堆字大小。Ex.233

通过erlang:trace/3跟踪标志garbage_collection和{tracer,dyntrace,[]}示踪模块可用。

例子：

gc_major_end: { cpu_id = 0 }, { pid = "<0.172.0>", reclaimed = 240, heap = 4185, old_heap = 0 }

1.4束迹点

所有跟踪点都在org_erlang_otp

所有Erlang类型都是LTTNG中等效的字符串。

scheduler_poll

scheduler : integer*调度程序ID。Ex.1

runnable : integer*可运行。Ex.1

例子：

scheduler_poll: { cpu_id = 4 }, { scheduler = 1, runnable = 1 }

driver_init

driver : string*驱动器名。Ex."efile"

major : integer*主要版本。Ex.3

minor : integer*次要版本。Ex.1

flags : integer*标志。Ex.1

例子：

driver_init: { cpu_id = 2 }, { driver = "caller_drv", major = 3, minor = 3, flags = 1 }

driver_start

pid : string*进程ID。Ex."<0.131.0>"

driver : string*驱动器名。Ex."efile"

port : string::端口ID。Ex."#Port<0.1031>"

例子：

driver_start: { cpu_id = 2 }, { pid = "<0.198.0>", driver = "caller_drv", port = "#Port<0.3676>" }

driver_output

pid : string*进程ID。Ex."<0.131.0>"

port : string::端口ID。Ex. "#Port<0.1031>"

driver : string::驱动程序名称。Ex."efile"

bytes : integer*返回的数据的大小。Ex.82

例子：

driver_output: { cpu_id = 2 }, { pid = "<0.198.0>", port = "#Port<0.3677>", driver = "/bin/sh -s unix:cmd", bytes = 36 }

driver_outputv

pid : string*进程ID。Ex."<0.131.0>"

port : string::端口ID。Ex."#Port<0.1031>"

driver : string::驱动程序名称。Ex."efile"

bytes : integer*返回的数据的大小。Ex.82

例子：

driver_outputv: { cpu_id = 5 }, { pid = "<0.194.0>", port = "#Port<0.3663>", driver = "tcp_inet", bytes = 3 }

driver_ready_input

pid : string*进程ID。Ex."<0.131.0>"

port : string::端口ID。Ex."#Port<0.1031>"

driver : string::驱动程序名称。Ex."efile"

例子：

driver_ready_input: { cpu_id = 5 }, { pid = "<0.189.0>", port = "#Port<0.3637>", driver = "inet_gethost 4 " }

driver_ready_output

pid : string*进程ID。Ex."<0.131.0>"

port : string::端口ID。Ex."#Port<0.1031>"

driver : string::驱动程序名称。Ex."efile"

例子：

driver_ready_output: { cpu_id = 5 }, { pid = "<0.194.0>", port = "#Port<0.3663>", driver = "tcp_inet" }

driver_timeout

pid : string*进程ID。Ex."<0.131.0>"

port : string::端口ID。Ex."#Port<0.1031>"

driver : string::驱动程序名称。Ex."efile"

例子：

driver_timeout: { cpu_id = 5 }, { pid = "<0.196.0>", port = "#Port<0.3664>", driver = "tcp_inet" }

driver_stop_select

driver : string::驱动程序名称 Ex."efile"例如：driver_stop_select：{cpu_id = 5}，{driver =“unknown”} driver_flush

pid : string*进程ID。Ex."<0.131.0>"

port : string::端口ID。Ex."#Port<0.1031>"

driver : string::驱动程序名称。Ex."efile"

例子：

driver_flush: { cpu_id = 7 }, { pid = "<0.204.0>", port = "#Port<0.3686>", driver = "tcp_inet" }

driver_stop

pid : string*进程ID。Ex."<0.131.0>"

port : string::端口ID。Ex."#Port<0.1031>"

driver : string::驱动程序名称。Ex."efile"

例子：

driver_stop: { cpu_id = 5 }, { pid = "[]", port = "#Port<0.3673>", driver = "efile" }

司机[医]加工过程[医]出口

pid : string*进程ID。Ex."<0.131.0>"

port : string::端口ID。Ex."#Port<0.1031>"

driver : string::驱动程序名称。Ex."efile"

driver_ready_async

pid : string*进程ID。Ex."<0.131.0>"

port : string::端口ID。Ex."#Port<0.1031>"

driver : string::驱动程序名称。Ex."efile"

例子：

driver_ready_async: { cpu_id = 3 }, { pid = "<0.181.0>", port = "#Port<0.3622>", driver = "efile" }

driver_call

pid : string*进程ID。Ex."<0.131.0>"

port : string::端口ID。Ex."#Port<0.1031>"

driver : string::驱动程序名称。Ex."efile"

command : integer*整数命令。前。1

bytes : integer*返回的数据的大小。前。82

例子：

driver_call: { cpu_id = 2 }, { pid = "<0.202.0>", port = "#Port<0.3676>", driver = "caller_drv", command = 0, bytes = 2 }

driver_control

pid : string*进程ID。Ex."<0.131.0>"

port : string::端口ID。Ex."#Port<0.1031>"

driver : string::驱动程序名称。Ex."efile"

command : integer::命令整数。Ex 1

bytes : integer::返回的数据大小。Ex 82

例子：

driver_control: { cpu_id = 3 }, { pid = "<0.32767.8191>", port = "#Port<0.0>", driver = "forker", command = 83, bytes = 32 }

aio_pool_get

port : string::端口ID。Ex "#Port<0.1031>"

length : integer::异步队列长度。Ex 0

例子：

aio_pool_get: { cpu_id = 4 }, { port = "#Port<0.3614>", length = 0 }

aio_pool_put

port : string::端口ID。Ex "#Port<0.1031>"

length : integer::异步队列长度。Ex -1

异步队列长度没有为put操作定义。

例子：

aio_pool_put: { cpu_id = 3 }, { port = "#Port<0.3614>", length = -1 }

carrier_create

type : string*载体类型。Ex "ets_alloc"

instance : integer*分配程序实例。Ex 1

size : integer*载体大小。Ex 262144

mbc_carriers : integer*例如多块运营商的数目。Ex 3

mbc_carriers_size : integer*例如，多块区块承运人的总规模。Ex1343488

mbc_blocks : integer*例如多块块的数目。Ex122

mbc_blocks_size : integer*例如，所有多块区块的总大小。Ex 285296

sbc_carriers : integer*例如单块运营商的数目。Ex 1

sbc_carriers_size : integer*例如，单块区块承运人的总规模。Ex 1343488

sbc_blocks : integer*例如单一块的数目。Ex 1

sbc_blocks_size : integer*例如所有单块区块的总大小。Ex 285296

例子：

carrier_create: { cpu_id = 2 }, { type = "ets_alloc", instance = 7, size = 2097152, mbc_carriers = 4, mbc_carriers_size = 3440640, mbc_blocks = 526, mbc_blocks_size = 1278576, sbc_carriers = 0, sbc_carriers_size = 0, sbc_blocks = 0, sbc_blocks_size = 0 }

carrier_destroy

type : string*载体类型。Ex "ets_alloc"

instance : integer*分配程序实例。Ex 1

size : integer*载体大小。Ex 262144

mbc_carriers : integer*例如多块运营商的数目。Ex 3

mbc_carriers_size : integer*例如，多块区块承运人的总规模。Ex 1343488

mbc_blocks : integer*例如多块块的数目。Ex 122

mbc_blocks_size : integer*例如，所有多块区块的总大小。Ex 285296

sbc_carriers : integer*例如单块运营商的数目。Ex 1

sbc_carriers_size : integer*例如，单块区块承运人的总规模。Ex 1343488

sbc_blocks : integer*例如单一块的数目。Ex 1

sbc_blocks_size : integer*例如所有单块区块的总大小。Ex 285296

例子：

carrier_destroy: { cpu_id = 6 }, { type = "ets_alloc", instance = 7, size = 262144, mbc_carriers = 3, mbc_carriers_size = 3178496, mbc_blocks = 925, mbc_blocks_size = 2305336, sbc_carriers = 0, sbc_carriers_size = 0, sbc_blocks = 0, sbc_blocks_size = 0 }

carrier_pool_put

type : string*载体类型。Ex "ets_alloc"

instance : integer*分配程序实例。Ex 1

size : integer*载体大小。Ex 262144

例子：

carrier_pool_put: { cpu_id = 3 }, { type = "ets_alloc", instance = 5, size = 1048576 }

carrier_pool_get

type : string*载体类型。Ex "ets_alloc"

instance : integer*分配程序实例。Ex 1

size : integer*载体大小。Ex 262144

例子：

carrier_pool_get: { cpu_id = 7 }, { type = "ets_alloc", instance = 4, size = 3208 }

1.5 过程跟踪实例

跟踪os_mon和朋友的过程示例。

在bash shell中清理lttng的开始。

$ lttng create erlang-demo
Spawning a session daemon
Session erlang-demo created.
Traces will be written in /home/egil/lttng-traces/erlang-demo-20160526-165920

启用lttng启动Erlang节点。

$ erl
Erlang/OTP 19 [erts-8.0] [source-4d7b24d] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [lttng]

Eshell V8.0  (abort with ^G)
1>

加载dyntrace模块。

1> l(dyntrace).
{module,dyntrace}

所有通过动态跟踪的跟踪点现在都是visi胆，可以通过lttng list -u...

为Erlang启用process_register LTTng跟踪点。

$ lttng enable-event -u org_erlang_dyntrace:process_register
UST event org_erlang_dyntrace:process_register created in channel channel0

为新流程启用流程跟踪并使用dyntrace作为追踪器后端。

2> erlang:trace(new,true,[procs,{tracer,dyntrace,[]}]).
0

开始LTTng跟踪。

$ lttng start
Tracing started for session erlang-demo

os_mon在Erlang中启动应用程序。

3> application:ensure_all_started(os_mon).
{ok,[sasl,os_mon]}

停止LTTng跟踪并查看结果。

$ lttng stop
Tracing stopped for session erlang-demo
$ lttng view
[17:20:42.561168759] (+?.?????????) elxd1168lx9 org_erlang_dyntrace:process_register: \
    { cpu_id = 5 }, { pid = "<0.66.0>", name = "sasl_sup", type = "register" }
[17:20:42.561215519] (+0.000046760) elxd1168lx9 org_erlang_dyntrace:process_register: \
    { cpu_id = 5 }, { pid = "<0.67.0>", name = "sasl_safe_sup", type = "register" }
[17:20:42.562149024] (+0.000933505) elxd1168lx9 org_erlang_dyntrace:process_register: \
    { cpu_id = 5 }, { pid = "<0.68.0>", name = "alarm_handler", type = "register" }
[17:20:42.571035803] (+0.008886779) elxd1168lx9 org_erlang_dyntrace:process_register: \
    { cpu_id = 5 }, { pid = "<0.69.0>", name = "release_handler", type = "register" }
[17:20:42.574939868] (+0.003904065) elxd1168lx9 org_erlang_dyntrace:process_register: \
    { cpu_id = 5 }, { pid = "<0.74.0>", name = "os_mon_sup", type = "register" }
[17:20:42.576818712] (+0.001878844) elxd1168lx9 org_erlang_dyntrace:process_register: \
    { cpu_id = 5 }, { pid = "<0.75.0>", name = "disksup", type = "register" }
[17:20:42.580032013] (+0.003213301) elxd1168lx9 org_erlang_dyntrace:process_register: \
    { cpu_id = 5 }, { pid = "<0.76.0>", name = "memsup", type = "register" }
[17:20:42.583046339] (+0.003014326) elxd1168lx9 org_erlang_dyntrace:process_register: \
    { cpu_id = 5 }, { pid = "<0.78.0>", name = "cpu_sup", type = "register" }
[17:20:42.586206242] (+0.003159903) elxd1168lx9 org_erlang_dyntrace:process_register: \
    { cpu_id = 5 }, { pid = "<0.82.0>", name = "timer_server", type = "register" }