作用
将http请求与响应的内容发到kafka的topic，以json的形式发送存储
配置相关
- log_format为自定义配置字段，添加后，默认的请求响应消息将被覆盖
{"_meta": {"disable": false},"batch_max_size": 1,"brokers": [{"host": "192.168.10.132","port": 9091},{"host": "192.168.10.133","port": 9092},{"host": "192.168.10.134","port": 9097}],"disable": false,"include_req_body": false,"include_resp_body": false,"kafka_topic": "apisix-logger-test","log_format": {"api": "$request_uri","userId": "$http_sub","userName": "$http_preferred_username","now":"$time_iso8601",},"producer_type": "async","required_acks": 1
}
$开头表示是系统变量$http_开头的，表示是从请求头里获取- 如果获取字段的内容为空，则不会出现在消息体里
- kafka里存储的消息为以下内容
{"now": "2024-09-10T02:01:22+00:00","route_id": "528226539821597458","api": "/pkulaw-chat-gpt-api-4/6.0.0.0/gpt/writer/ping",
}
上面配置，是直接在路由或者全局插件配置中，可以通过apisix-dashboard进行配置，使用log_format对记录的信息进行了设置，我们不建议自定义log_format，因为默认的就是最全的信息
不使用log_format，默认的消息体内容如下
[{"client_ip": "192.168.60.136","upstream": "10.42.4.236:8080","apisix_latency": 7.0000324249268,"start_time": 1716198064095,"latency": 16.000032424927,"request": {"uri": "/kpi/hello","method": "GET","size": 1757,"url": "http://test-apisix.pkulaw.com:9080/kpi/hello","headers": {"accept": "text/html,application/xhtml+xml,application/xml;q\u003d0.9,image/avif,image/webp,*/*;q\u003d0.8","connection": "close","user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0","cookie": "Hm_lvt_8266968662c086f34b2a3e2ae9014bf8\u003d1715752532; Hm_up_8266968662c086f34b2a3e2ae9014bf8\u003d%7B%22ysx_yhqx_20220602%22%3A%7B%22value%22%3A%220%22%2C%22scope%22%3A1%7D%2C%22ysx_hy_20220527%22%3A%7B%22value%22%3A%2206%22%2C%22scope%22%3A1%7D%2C%22uid_%22%3A%7B%22value%22%3A%22fd9959db-ab28-eb11-b390-00155d3c0709%22%2C%22scope%22%3A1%7D%2C%22ysx_yhjs_20220602%22%3A%7B%22value%22%3A%221%22%2C%22scope%22%3A1%7D%7D; cookieUUID\u003dcookieUUID_1715593171791; Hm_lpvt_8266968662c086f34b2a3e2ae9014bf8\u003d1715852645; CookieId\u003daa4be28db45fcd6c3edd84d304533838; SUB\u003d79781f2d-8b4f-483e-ac31-dd55ada12e21; preferred_username\u003dphone2021090709182413393; session_state\u003dde594c09-5681-47c2-ad76-6a7979ff38cd; sensorsdata2015jssdkcross\u003d%7B%22distinct_id%22%3A%22CPcsuEJK%22%2C%22first_id%22%3A%2218f938551ccfe-0b472eec538364-e505625-2073600-18f938551cde56%22%2C%22props%22%3A%7B%22%24latest_traffic_source_type%22%3A%22%E7%9B%B4%E6%8E%A5%E6%B5%81%E9%87%8F%22%2C%22%24latest_search_keyword%22%3A%22%E6%9C%AA%E5%8F%96%E5%88%B0%E5%80%BC_%E7%9B%B4%E6%8E%A5%E6%89%93%E5%BC%80%22%2C%22%24latest_referrer%22%3A%22%22%7D%2C%22%24device_id%22%3A%2218f938551ccfe-0b472eec538364-e505625-2073600-18f938551cde56%22%7D; sajssdk_2015_cross_new_user\u003d1","sec-fetch-dest": "document","upgrade-insecure-requests": "1","accept-language": "zh-CN,en;q\u003d0.8,en-US;q\u003d0.7,zh;q\u003d0.5,zh-TW;q\u003d0.3,zh-HK;q\u003d0.2","sec-fetch-mode": "navigate","accept-encoding": "gzip, deflate, br","sec-fetch-site": "none","sec-fetch-user": "?1","host": "test-apisix.pkulaw.com","x-forwarded-for": "111.198.143.194"},"querystring": {}},"response": {"status": 200,"headers": {"server": "APISIX/3.8.0","content-type": "application/json","x-ratelimit-limit": "5","transfer-encoding": "chunked","connection": "close","date": "Mon, 20 May 2024 09:41:04 GMT","x-ratelimit-reset": "60","x-ratelimit-remaining": "4"},"size": 2064},"route_id": "513923429800346372","upstream_latency": 9,"service_id": "","server": {"version": "3.8.0","hostname": "apisix-78bcfb45c6-26746"}}
]在log_format的同步添加扩展字段
- 这需要修改kafka-logger的源码了
- 获取user-agent中的操作系统，浏览器等信息，你可以添加下面的方法
local function parse_user_agent(user_agent)local os, browser, version-- 检测操作系统if user_agent:find("Windows") thenos = "Windows"elseif user_agent:find("Macintosh") thenos = "Mac OS"elseif user_agent:find("Linux") thenos = "Linux"elseif user_agent:find("Android") thenos = "Android"elseif user_agent:find("iPhone") thenos = "iOS"end-- 检测浏览器if user_agent:find("Chrome") thenbrowser = "Chrome"version = user_agent:match("Chrome%/(%d+%.%d+)")elseif user_agent:find("Firefox") thenbrowser = "Firefox"version = user_agent:match("Firefox%/(%d+%.%d+)")elseif user_agent:find("Safari") thenbrowser = "Safari"version = user_agent:match("Version%/(%d+%.%d+)")elseif user_agent:find("MSIE") thenbrowser = "Internet Explorer"version = user_agent:match("MSIE (%d+%.%d+)")elseif user_agent:find("Trident") thenbrowser = "Internet Explorer"version = user_agent:match("rv:(%d+%.%d+)")endreturn {os = os or "Unknown",browser = browser or "Unknown",version = version or "Unknown"}
end在kafka-logger的_M.log(conf, ctx)方法中，添加浏览器扩展字段
function _M.log(conf, ctx)local entryif conf.meta_format == "origin" thenentry = log_util.get_req_original(ctx, conf)elseentry = log_util.get_log_entry(plugin_name, conf, ctx)end-- 添加扩展字段开始local user_agent = ngx.var.http_user_agentlocal info = parse_user_agent(user_agent)entry.os = info.osentry.browser=info.browserentry.version=info.version-- 添加扩展字段结束if batch_processor_manager:add_entry(conf, entry) thenreturnend
...
