Nginx实战 基于Lua语言的配置、开发与架构详解

"1.业内专家刘凯、冯浩、屈耀华、杨明翰、彭赫联袂力荐。 2.本书深入浅出地讲解了Nginx+Lua在实战场景中的各种使用技巧和方法，涉及配置、模块、缓存、日志分析、静态容灾、反向代理、爬虫、性能分析与优化等众多方面，帮助提升服务性能。 3.所有实战案例都有配套源码，方便读者学习和使用。 "

本书主要讲解了Nginx 在反向代理和应用开发中的作用，阅读本书可以了解Nginx 在互联网开发中扮演的多个角色，充分利用这些角色的各项功能有助于提升服务的整体性能。本书所介绍的大部分功能是通过Nginx+Lua 进行开发和配置的，但并不要求读者精通Lua，在必要的位置，本书会对Lua 进行选择性讲解。涉及实战的内容会有配套源码，方便读者学习和使用。本书适合广大运维人员和开发人员学习，对使用Nginx 完成各种服务架构感兴趣的架构师也可以阅读本书。阅读本书需要有对Nginx 的初级或中级配置经验。

"王力 折800运维架构师，Nginx开发工程师，具有多年互联网相关项目的开发经验。喜欢使用小众技术完成运维、开发和测试任务，擅长通过架构设计降低开发难度，擅长搭建Nginx高性能服务。 汤永全 折800运维主管，具有多年互联网相关项目的运维工作经验。历经传统运维和互联网运维，一步一个脚印，积累了不少运维相关的经验和感悟。活跃于成都运维管理人员的线下组织“运维罗汉堂”。 "

章 Nginx 学前必知................................................................................................................................. 1 1.1 HTTP 请求报文................................................................................................................................. 1 1.2 HTTP 响应报文................................................................................................................................. 2 1.3 安装Nginx......................................................................................................................................... 2 1.4 支持HTTPS....................................................................................................................................... 4 1.5 添加模块............................................................................................................................................ 4 1.6 小结..................................................................................................................................................... 4 第2 章 基础配置............................................................................................................................................. 5 2.1 Nginx 指令和指令块........................................................................................................................ 5 2.2 Nginx 基本配置说明........................................................................................................................ 6 2.2.1 main 配置.................................................................................................................................. 6 2.2.2 与客户端有关的配置............................................................................................................... 7 2.2.3 server 块..................................................................................................................................... 7 2.2.4 location 块.................................................................................................................................. 8 2.3 include 的使用................................................................................................................................... 9 2.4 常见配置............................................................................................................................................ 9 2.4.1 常见配置注解..........................................................................................................................10 2.4.2 常见配置实战技巧..................................................................................................................11 2.5 内置变量...........................................................................................................................................13 2.5.1 常见内置变量..........................................................................................................................13 2.5.2 常见内置变量实战技巧...........................................................................................................15 2.6 小结....................................................................................................................................................16 第3 章 强化基础配置....................................................................................................................................17 3.1 牢记Context .....................................................................................................................................17 3.2 获取请求的IP 地址........................................................................................................................18 3.2.1 获取用户的真实IP 地址.........................................................................................................18 3.2.2 防止IP 地址伪造.....................................................................................................................19 3.2.3 后端服务器对IP 地址的需求.................................................................................................19 3.3 管理请求的行为..............................................................................................................................20 3.3.1 IP 地址的访问.................................................................................................................20 3.3.2 auth 身份验证...........................................................................................................................21 3.3.3 利用LDAP 服务加强安全......................................................................................................22 3.3.4 satisfy 二选一的访问功能...............................................................................................23 3.4 proxy 代理.........................................................................................................................................23 3.4.1 proxy_pass 请求代理规则........................................................................................................24 3.4.2 减少后端服务器的网络开销...................................................................................................24 3.4.3 控制请求头和请求体..............................................................................................................25 3.4.4 控制请求和后端服务器的交互时间.......................................................................................26 3.5 upstream 使用手册..........................................................................................................................26 3.5.1 代理多台服务器......................................................................................................................27 3.5.2 故障转移..................................................................................................................................28 3.5.3 负载均衡..................................................................................................................................29 3.5.4 通过hash 分片提升缓存命中率..............................................................................................29 3.5.5 利用长连接提升性能..............................................................................................................30 3.5.6 利用resolver 加速对内部域名的访问....................................................................................31 3.6 rewrite 使用手册..............................................................................................................................32 3.6.1 内部重定向..............................................................................................................................32 3.6.2 域名跳转..................................................................................................................................33 3.6.3 跳转POST 请求.......................................................................................................................34 3.6.4 设置变量的值..........................................................................................................................34 3.7 限速白名单.......................................................................................................................................35 3.8 日志....................................................................................................................................................36 3.8.1 记录自定义变量......................................................................................................................36 3.8.2 日志格式规范..........................................................................................................................36 3.8.3 日志存储..................................................................................................................................37 3.9 HTTP 执行阶段................................................................................................................................38 3.10 小结..................................................................................................................................................39 第4 章 常用模块精解....................................................................................................................................40 4.1 定制HTTP 头信息..........................................................................................................................40 4.1.1 使用ngx_http_headers_module 设置响应头...........................................................................40 4.1.2 使用headers-more-nginx 控制请求头和响应头.....................................................................43 4.2 第三方模块set-misc-nginx............................................................................................................45 4.2.1 设置变量..................................................................................................................................46 4.2.2 防止SQL 注入.........................................................................................................................46 4.2.3 字符串非转义和转义..............................................................................................................47 4.2.4 基于键值的集群分片..............................................................................................................48 4.2.5 base 编码..................................................................................................................................48 4.2.6 md5 编码..................................................................................................................................50 4.2.7 生成随机数..............................................................................................................................50 4.2.8 本地时间的输出......................................................................................................................52 4.2.9 实战经验..................................................................................................................................52 4.3 图片的处理.......................................................................................................................................53 4.3.1 image_filter 图片处理..............................................................................................................53 4.3.2 采用渐进式方式打开JPEG 图片............................................................................................55 4.3.3 WebP 格式................................................................................................................................56 4.3.4 优化图片..................................................................................................................................56 4.3.5 实战经验：动态切图..............................................................................................................58 4.4 TCP 和UDP 代理............................................................................................................................58 4.4.1 代理配置说明..........................................................................................................................58 4.4.2 DNS 服务的反向代理..............................................................................................................62 4.4.3 MySQL 集群代理配置.............................................................................................................62 4.4.4 实战经验..................................................................................................................................63 4.5 常用模块介绍...................................................................................................................................63 4.5.1 基于访问IP 地址跳转到对应城市.........................................................................................63 4.5.2 修改响应内容..........................................................................................................................65 4.5.3 零像素文件的生成及其作用...................................................................................................66 4.5.4 图片的防盗链..........................................................................................................................67 4.6 小结....................................................................................................................................................68 第5 章 缓存系统............................................................................................................................................69 5.1 缓存配置说明...................................................................................................................................69 5.2 控制缓存有效期..............................................................................................................................71 5.3 性能优化...........................................................................................................................................72 5.3.1 缓存未命中的很好实践...........................................................................................................72 5.3.2 横向扩展很好实践..................................................................................................................75 5.3.3 避免硬盘I/O 阻塞...................................................................................................................76 5.3.4 集群模式..................................................................................................................................77 5.4 高可用方案.......................................................................................................................................77 5.5 proxy_cache 配置模板....................................................................................................................78 5.6 小结....................................................................................................................................................81 第6 章 引入Lua.............................................................................................................................................82 6.1 引入Lua 的原因..............................................................................................................................82 6.2 Lua 和LuaJIT ...................................................................................................................................83 6.3 环境搭建...........................................................................................................................................83 6.4 Lua 的数据类型...............................................................................................................................84 6.4.1 类型说明..................................................................................................................................84 6.4.2 类型示例..................................................................................................................................85 6.5 表达式................................................................................................................................................89 6.5.1 算术运算符..............................................................................................................................89 6.5.2 关系运算符..............................................................................................................................90 6.5.3 逻辑运算符..............................................................................................................................91 6.5.4 字符串连接和字符串长度计算...............................................................................................92 6.5.5 运算符优先级..........................................................................................................................93 6.6 变量....................................................................................................................................................93 6.6.1 全局变量..................................................................................................................................94 6.6.2 局部变量..................................................................................................................................94 6.6.3 变量赋值..................................................................................................................................94 6.7 流程控制...........................................................................................................................................95 6.7.1 if-else ........................................................................................................................................95 6.7.2 for 循环.....................................................................................................................................96 6.7.3 while 循环.................................................................................................................................97 6.7.4 break 和return ..........................................................................................................................97 6.8 函数....................................................................................................................................................98 6.8.1 函数格式..................................................................................................................................98 6.8.2 传参方式..................................................................................................................................99 6.8.3 函数的创建位置....................................................................................................................100 6.9 模块..................................................................................................................................................100 6.9.1 模块格式................................................................................................................................101 6.9.2 加载模块................................................................................................................................101 6.10 Lua 常见操作...............................................................................................................................102 6.10.1 操作table .............................................................................................................................102 6.10.2 定义字符串..........................................................................................................................103 6.10.3 字符串连接..........................................................................................................................104 6.11 引入Lua 的插曲..........................................................................................................................104 6.12 小结................................................................................................................................................105 第7 章 Lua-Nginx-Module 常用指令......................................................................................................106 7.1 Nginx 和OpenResty ......................................................................................................................106 7.2 安装Ngx_Lua ................................................................................................................................107 7.3 牢记Context ...................................................................................................................................108 7.4 Hello World .....................................................................................................................................108 7.5 避免I/O 阻塞..................................................................................................................................109 7.6 定义模块搜索路径........................................................................................................................109 7.6.1 定义Lua 模块的搜索路径.....................................................................................................109 7.6.2 定义C 模块的搜索路径........................................................................................................110 7.7 读/写Nginx 的内置变量..............................................................................................................110 7.8 控制请求头.....................................................................................................................................111 7.8.1 添加请求头............................................................................................................................111 7.8.2 清除请求头............................................................................................................................112 7.8.3 获取请求头............................................................................................................................112 7.9 控制响应头.....................................................................................................................................113 7.9.1 获取响应头............................................................................................................................113 7.9.2 修改响应头............................................................................................................................114 7.9.3 清除响应头............................................................................................................................116 7.10 读取请求体...................................................................................................................................116 7.10.1 强制获取请求体..................................................................................................................116 7.10.2 用同步非阻塞方式获取请求体...........................................................................................117 7.10.3 使用场景示例......................................................................................................................118 7.10.4 使用建议..............................................................................................................................121 7.11 输出响应体...................................................................................................................................121 7.11.1 异步发送响应体..................................................................................................................121 7.11.2 同步发送响应体..................................................................................................................122 7.12 正则表达式...................................................................................................................................124 7.12.1 单一捕获..............................................................................................................................124 7.12.2 全部捕获..............................................................................................................................125 7.12.3 更高效的匹配和捕获..........................................................................................................126 7.12.4 替换数据..............................................................................................................................128 7.12.5 转义符号..............................................................................................................................129 7.13 子请求............................................................................................................................................130 7.13.1 请求方法..............................................................................................................................130 7.13.2 单一子请求..........................................................................................................................130 7.13.3 并发子请求..........................................................................................................................134 7.14 获取Nginx 的环境变量.............................................................................................................135 7.14.1 获取环境所在的模块..........................................................................................................135 7.14.2 确认调试模式......................................................................................................................136 7.14.3 获取prefix 路径...................................................................................................................136 7.14.4 获取Nginx 的版本号...........................................................................................................136 7.14.5 获取configure 信息.............................................................................................................136 7.14.6 获取Ngx_Lua 的版本号......................................................................................................137 7.14.7 判断worker 进程是否退出.................................................................................................137 7.14.8 获取worker 进程的ID ........................................................................................................137 7.14.9 获取worker 进程的数量.....................................................................................................137 7.15 定时任务.......................................................................................................................................138 7.15.1 创建定时任务......................................................................................................................138 7.15.2 性能优化..............................................................................................................................140 7.15.3 禁用的Lua API....................................................................................................................141 7.16 常用指令.......................................................................................................................................142 7.16.1 请求重定向..........................................................................................................................142 7.16.2 日志记录..............................................................................................................................144 7.16.3 请求中断处理......................................................................................................................146 7.17 提升开发和测试效率.................................................................................................................149 7.17.1 断开客户端连接..................................................................................................................149 7.17.2 请求休眠..............................................................................................................................150 7.17.3 获取系统时间......................................................................................................................150 7.17.4 编码与解码..........................................................................................................................152 7.17.5 防止SQL 注入.....................................................................................................................154 7.17.6 判断是否为子请求..............................................................................................................155 7.17.7 设置MIME 类型..................................................................................................................156 7.18 小结................................................................................................................................................156 第8 章 Ngx_Lua 的执行阶段....................................................................................................................157 8.1 init_by_lua_block...........................................................................................................................157 8.1.1 阶段说明................................................................................................................................157 8.1.2 初始化配置............................................................................................................................158 8.1.3 控制初始值............................................................................................................................159 8.1.4 init_by_lua_file.......................................................................................................................160 8.1.5 可使用的Lua API 指令.........................................................................................................160 8.2 init_worker_by_lua_block ............................................................................................................160 8.2.1 阶段说明................................................................................................................................160 8.2.2 启动Nginx 的定时任务.........................................................................................................161 8.2.3 动态进行后端健康检查.........................................................................................................162 8.3 set_by_lua_block............................................................................................................................165 8.3.1 阶段说明................................................................................................................................165 8.3.2 变量赋值................................................................................................................................165 8.3.3 rewrite 阶段的混用模式........................................................................................................166 8.3.4 阻塞事件................................................................................................................................167 8.3.5 被禁用的Lua API 指令.........................................................................................................167 8.4 rewrite_by_lua_block ....................................................................................................................168 8.4.1 阶段说明................................................................................................................................168 8.4.2 利用rewrite_by_lua_no_postpone 改变执行顺序.................................................................168 8.4.3 阶段控制................................................................................................................................169 8.5 access_by_lua_block......................................................................................................................169 8.5.1 阶段说明................................................................................................................................169 8.5.2 利用access_by_lua_no_postpone 改变执行顺序..................................................................170 8.5.3 阶段控制................................................................................................................................170 8.5.4 动态配置黑白名单................................................................................................................170 8.6 content_by_lua_block....................................................................................................................170 8.6.1 阶段说明................................................................................................................................170 8.6.2 动态调整执行文件的路径.....................................................................................................171 8.7 balancer_by_lua_block..................................................................................................................171 8.7.1 阶段说明................................................................................................................................171 8.7.2 被禁用的Lua API 指令.........................................................................................................172 8.8 header_filter_by_lua_block ..........................................................................................................172 8.8.1 阶段说明................................................................................................................................172 8.8.2 被禁用的Lua API 指令.........................................................................................................173 8.9 body_filter_by_lua_block .............................................................................................................173 8.9.1 阶段说明................................................................................................................................173 8.9.2 控制响应体数据....................................................................................................................173 8.9.3 被禁用的Lua API 指令.........................................................................................................175 8.10 log_by_lua_block.........................................................................................................................176 8.10.1 阶段说明..............................................................................................................................176 8.10.2 被禁用的Lua API 指令.......................................................................................................176 8.11 Lua 和ngx.ssl ...............................................................................................................................177 8.12 Ngx_Lua 执行阶段......................................................................................................................177 8.13 小结................................................................................................................................................180 第9 章 Nginx 与数据库的交互.................................................................................................................181 9.1 安装cjson........................................................................................................................................181 9.2 与MySQL 交互.............................................................................................................................183 9.2.1 安装lua-resty-mysql 模块......................................................................................................183 9.2.2 读取MySQL 数据.................................................................................................................183 9.2.3 执行多条SQL 语句...............................................................................................................187 9.2.4 防止SQL 注入.......................................................................................................................189 9.3 与Redis 交互..................................................................................................................................189 9.3.1 安装lua-resty-redis ................................................................................................................189 9.3.2 读/写Redis .............................................................................................................................189 9.3.3 管道命令................................................................................................................................191 9.3.4 密码登录................................................................................................................................193 9.3.5 其他执行命令........................................................................................................................194 9.4 与数据库交互的常见问题...........................................................................................................194 9.4.1 连接池....................................................................................................................................194 9.4.2 读/写分离...............................................................................................................................197 9.4.3 分离配置文件和代码............................................................................................................197 9.5 小结..................................................................................................................................................198 0 章 缓存利器........................................................................................................................................199 10.1 worker 进程的共享内存............................................................................................................200 10.1.1 创建共享内存区域..............................................................................................................200 10.1.2 操作共享内存......................................................................................................................201 10.1.3 制造消息队列......................................................................................................................205 10.1.4 lua-resty-core ........................................................................................................................207 10.1.5 配置环境..............................................................................................................................208 10.2 Lua 模块下的共享内存..............................................................................................................209 10.2.1 安装lua-resty-lrucache.........................................................................................................209 10.2.2 使用lua-resty-lrucache 进行缓存的方法............................................................................209 10.3 当前请求在各执行阶段间的数据共享...................................................................................213 10.3.1 ngx.ctx 的使用......................................................................................................................213 10.3.2 子请求和内部重定向的缓存区别.......................................................................................214 10.4 利用共享内存配置动态IP 地址认证.....................................................................................215 10.5 缓存和数据库的交互.................................................................................................................218 10.5.1 从数据库获取数据..............................................................................................................218 10.5.2 避免出现因缓存失效引起的“风暴”...............................................................................223 10.6 小结................................................................................................................................................228 1 章 动态管理upstream.....................................................................................................................229 11.1 实战需求分析...............................................................................................................................230 11.2 ngx_http_dyups_module .............................................................................................................230 11.2.1 安装ngx_http_dyups_module..............................................................................................230 11.2.2 动态管理upstream...............................................................................................................230 11.2.3 确保upstream 数据的完整性..............................................................................................232 11.3 nginx-upsync-module ..................................................................................................................233 11.3.1 安装nginx-upsync-module 和Consul..................................................................................233 11.3.2 Consul 的键值操作..............................................................................................................234 11.3.3 动态管理upstream...............................................................................................................235 11.3.4 验证动态配置功能..............................................................................................................237 11.3.5 高可用、高并发设计..........................................................................................................237 11.4 基于balancer_by_lua_block 的灵活控制...............................................................................238 11.5 小结................................................................................................................................................239 2 章 Nginx 日志分析系统...................................................................................................................240 12.1 实战需求分析..............................................................................................................................240 12.2 ngxtop 实时分析..........................................................................................................................241 12.3 Flume 方案的日志分析..............................................................................................................243 12.4 智能化nginx_log_analysis ........................................................................................................244 12.4.1 架构重构..............................................................................................................................244 12.4.2 日志远程传输......................................................................................................................245 12.4.3 时序数据库..........................................................................................................................245 12.4.4 日志规则设计......................................................................................................................245 12.5 lua-resty-logger-socket 传输方案..............................................................................................246 12.5.1 安装lua-resty-logger-socket.................................................................................................246 12.5.2 远程传输配置......................................................................................................................247 12.5.3 参数解读..............................................................................................................................248 12.6 时序数据库InfluxDB.................................................................................................................249 12.6.1 安装InfluxDB ......................................................................................................................249 12.6.2 基本概念和操作..................................................................................................................249 12.6.3 数据分析之查询函数..........................................................................................................250 12.6.4 数据存放之保留策略..........................................................................................................251 12.6.5 定时任务之连续查询..........................................................................................................251 12.6.6 客户端操作之API...............................................................................................................252 12.6.7 使用UDP 模式传输数据.....................................................................................................253 12.7 利用lua-resty-http 实现API 交互............................................................................................254 12.7.1 安装lua-resty-http................................................................................................................254 12.7.2 使用方式..............................................................................................................................254 12.8 提升InfluxDB 性能....................................................................................................................255 12.9 小结................................................................................................................................................255 3 章 静态容灾系统...............................................................................................................................256 13.1 荆棘之路.......................................................................................................................................257 13.2 设计之路.......................................................................................................................................259 13.3 架构流程图...................................................................................................................................261 13.3.1 反向代理系统......................................................................................................................261 13.3.2 日志分析系统......................................................................................................................261 13.3.3 后台系统..............................................................................................................................261 13.3.4 爬虫系统..............................................................................................................................262 13.3.5 容灾的缓存系统..................................................................................................................262 13.3.6 时间版本的用途..................................................................................................................263 13.3.7 异地容灾..............................................................................................................................263 13.4 核心代码解说..............................................................................................................................264 13.4.1 Ngx_Lua 应用......................................................................................................................264 13.4.2 爬虫和日志系统的关系.......................................................................................................266 13.4.3 全部容灾和部分容灾功能...................................................................................................266 13.5 静态容灾的智能关闭方案........................................................................................................267 13.5.1 从日志分析系统中复制请求...............................................................................................267 13.5.2 利用goreplay 复制流量.......................................................................................................267 13.5.3 Nginx 的镜像功能................................................................................................................268 13.5.4 灰度验证容灾系统缓存.......................................................................................................269 13.6 小结................................................................................................................................................269 4 章 深入挖掘反向代理......................................................................................................................270 14.1 验证码防御中心..........................................................................................................................270 14.2 鉴权管理中心..............................................................................................................................272 14.2.1 利用auth_request 管理鉴权................................................................................................272 14.2.2 利用Ngx_Lua 子请求实现鉴权功能..................................................................................273 14.3 并行访问.......................................................................................................................................274 14.3.1 轻线程的启动和终止..........................................................................................................275 14.3.2 等待和终止轻线程..............................................................................................................276 14.3.3 URL 的外部合并和内部并发..............................................................................................278 14.3.4 使用cosocket 实现外部访问...............................................................................................281 14.4 小结................................................................................................................................................281 5 章 爬虫.................................................................................................................................................282 15.1 区分搜索引擎爬虫和恶意爬虫................................................................................................282 15.2 应对搜索引擎爬虫......................................................................................................................284 15.2.1 搜索引擎的User-Agent.......................................................................................................284 15.2.2 Robots 协议..........................................................................................................................285 15.2.3 控制搜索引擎爬虫实战.......................................................................................................286 15.3 应对恶意爬虫..............................................................................................................................288 15.3.1 发现恶意爬虫......................................................................................................................288 15.3.2 抵御恶意爬虫之禁止访问...................................................................................................289 15.3.3 抵御恶意爬虫之验证码拦截...............................................................................................290 15.4 小插曲――使用假数据迷惑恶意爬虫...................................................................................290 15.5 小结................................................................................................................................................291 6 章 性能分析和优化...........................................................................................................................292 16.1 性能分析场景搭建......................................................................................................................292 16.1.1 安装SystemTap ...................................................................................................................292 16.1.2 LuaJIT 的Debug 模式..........................................................................................................293 16.1.3 开启PCRE 的Debug 模式..................................................................................................294 16.1.4 分析工具下载......................................................................................................................294 16.1.5 找出不支持Debug 模式的lib 库........................................................................................295 16.2 流量复制.......................................................................................................................................295 16.3 各项指标分析和优化建议........................................................................................................295 16.3.1 连接池使用状态分析..........................................................................................................295 16.3.2 找出读/写频繁的文件.........................................................................................................297 16.3.3 执行阶段耗时分析..............................................................................................................297 16.3.4 HTTP 连接数和文件打开数分析........................................................................................298 16.3.5 找出CPU“偷窃者”..........................................................................................................298 16.3.6 正则表达式耗时分析..........................................................................................................299 16.3.7 找出消耗CPU 资源较多的指令.........................................................................................301 16.3.8 利用火焰图展示和分析数据...............................................................................................303 16.4 检查全局变量..............................................................................................................................305 16.5 小结................................................................................................................................................305 7 章 值得拥有的OpenResty.............................................................................................................306 17.1 OPM...............................................................................................................................................307 17.2 使用DNS 提升访问效率...........................................................................................................309 17.3 TCP 和UDP 服务........................................................................................................................310 17.4 多层级缓存...................................................................................................................................312 17.5 lua-resty-core 扩展.......................................................................................................................313 17.5.1 字符串分割..........................................................................................................................313 17.5.2 Nginx 进程管理....................................................................................................................313 17.6 全局专享标识符UUID..............................................................................................................315 17.7 “全家福”awesome-resty ........................................................................................................316 17.8 OpenResty，未来！....................................................................................................................316 8 章 开发环境下的常见问题..............................................................................................................317 18.1 被截断的响应体..........................................................................................................................317 18.2 “邪恶”的if ...............................................................................................................................317 18.3 “贪婪”的正则匹配.................................................................................................................318 18.4 规范HTTP 状态码......................................................................................................................319 18.5 规范URL......................................................................................................................................319 18.6 proxy_set_header 的误操作.......................................................................................................320 18.7 开发环境下的证书问题.............................................................................................................320 18.8 深层次的错误重定向.................................................................................................................323 18.9 压测环境下的限速和短连接....................................................................................................323 18.10 小结..............................................................................................................................................323