Les Développeurs et les Performances Java

Début 2014, Eric Lefevre-Ardant m’a invité à enregistrer une interview sur son podcast “Une Tasse de Thé”. À cette époque-là, les performances Java étaient mon sujet de prédilection et nous avons donc échangé sur ce thème.

Hélas, ce podcast n’est plus en ligne. Éric m’a donc gentiment transmis l’audio, que je publie ci-dessous.

À l’époque, j’avais partagé sur ce blog une présentation qui regroupe concepts et outils que j’évoque dans le podcast. J’ai aussi publié un article sur le blog de Xebia qui décrit comment nous avons mis en place, dans l’équipe où nous étions, des tests de performance à moindre coût.

Posted in Uncategorized | Tagged , , , , , | Leave a comment

PXE booting

TL;DR The code used here is available on Github.

This article describes the steps I walked through in order to install an Ubuntu server on my Dell PowerEdge 110T server using Preboot eXecution Environment (PXE).

One may ask: why bother with PXE for ocasional installations? Wouldn’t be simpler to just use a bootable USB memory stick?

Well, from a practical point-of-view the answer is yes. Although, I decided to go through this process mainly for 2 reasons:

  1. as a lazy developer, I challenge myself to automate everything whenever possible
  2. I always wanted to do this, either professionally, or as a learning exercise (it looks like Iron man, and that’s cool)

That said, before we get started, it is important to understand some concepts and technologies, and how the whole thing works together.

PXE mechanics

Preboot eXecution Environment (PXE, pronounced as pixie) specification describes a standardized client-server environment that boots a software assembly, retrieved from the network, on PXE-enabled clients. – Wikipedia

On the client side it requires only a PXE-capable network interface controller (NIC), and uses a small set of industry-standard network protocols such as Dynamic Host Configuration Protocol (DHCP) and Trivial File Transfer Protocol (TFTP). In many computers, the PXE capability can be enabled in the BIOS.

When a machine is booted in PXE mode, it tries to acquire an IP address during the boot phase through DHCP, and it sends some PXE-specific options in its DHCPDISCOVER packet. A PXE-enabled DHCP server will then respond to the client’s request with a DHCPOFFER packet containing not only default network information (i.e. IP address), but also the TFTP server IP address and the name of the Network Bootstrap Program (NBP), in this case pxelinux.0. The client can then download the NBP from the TFTP server and start the boot process. More information about DHCP sessions here.

Once NBP is booted, it will load the configuration from pxelinux.cfg/default in order to print the installation menu or just proceed with the installation, following what has been configured. The installation menu configuration also specifies the initrd image that will handle the installation process. Check the Syslinux Project wiki for more details about the PXE boot process.

Although, I’d like to keep using my router as my default DHCP server (setup a second DHCP server just for PXE sounds like a bit of waste in my case). The problem is that my router does not have any PXE capability.

In order to address this issue, there is something called proxyDHCP, which is a second DHCP server which only provides PXE-specific information to the PXE client through DHCP.

BIOS x UEFI

The Basic Input/Output System (BIOS) is a firmware that comes pre-installed on computers, and sits between hardware and software.

Once the machine is turned on, the BIOS is responsible for initialising and configuring all the components, like CPU, memory, disks, etc. If the BIOS was not there, we’d probably be using jumpers to handle all the configuration. After the initialisation, the BIOS will perform the power-on self test (POST), to check whether all the components are working properly. Finally, the BIOS will look for bootable devices in order to hand off control to the operating system (OS).

The BIOS has been there for a long time and has some serious limitations. The BIOS can only handle Master Boot Records (MBR) until roughly 2 terabytes, for example. Slow boot time, lack of mouse support and poor graphics also figure in the list of BIOS limitations.

The Unified Extensible Firmware Interface (UEFI) is the successor of BIOS. The UEFI aims to fix BIOS limitations and add more features, like looking for malware during the boot process, adding some capabilities for remote troubleshooting and configuration, and more.

That said, it is important to know which boot mechanism is being used. A PXE server configured to work only with BIOS won’t work with UEFI, and the other way round. A netboot image is commonly prepared having one or another of these two mechanisms in mind. This article describes a PXE server configuration to be used with BIOS.

Setup a PXE server

In order to setup a PXE server, the following tools and techniques were used:

  • Vagrant/VirtualBox/Ubuntu server: in order to avoid installing tools I won’t be using everyday on my machine, and to be able to re-use the same mechanism in another environment
  • Dnsmasq: in order to setup a proxyDHCP and a TFTP server
  • Preseeding: in order to spare myself from manually installing the OS

Setup a Vagrant box

The first step is to configure a Vagrant box running Ubuntu server over VirtualBox. At the network level, the box will be configured in bridge mode. In practice, this will leave the Virtual Machine (VM) running at the same level as the machine hosting it, allowing it to communicate directly with the DHCP server, and sparing some Network Address Translation (NAT) configuration.

Here’s the Vagrantfile that configures a box in bridge mode:

Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/xenial64"
config.vm.network "public_network"
end

Based on the Vagrantfile, the command bellow will download an Ubuntu box from a HashiCorp’s catalogue, apply some configuration (if any) and start the VM.

$ vagrant up

Since this configuration is using bridge networking through Vagrant public_network, it is necessary to inform Vagrant which network it should use in the bridge. This will be the network interface the host machine uses to connect to the DHCP server and, consequently, the internet (in this case en0).

default: Available bridged network interfaces:
1) en0: Wi-Fi (AirPort)
2) en1: Thunderbolt 1
3) en2: Thunderbolt 2
4) bridge0
5) p2p0
6) awdl0
default: When choosing an interface, it is usually the one that is
default: being used to connect to the internet.
default: Which interface should the network bridge to?

Download the Ubuntu Netboot Image

A netboot image is a bootable image optimised to perform an installation via network (as opposed to an ISO image). It generally contains:

  1. a small network boot manager program like pxelinux.0 (or NBP), which is used to deploy the installation menu and load a full installation image;
  2. and a second-stage bootable full image (typically initrd), used to perform the installation

Download an Ubuntu netboot image that will handle the installation process over the internet:

$  wget http://archive.ubuntu.com/ubuntu/dists/xenial/main/installer-amd64/current/images/netboot/netboot.tar.gz \
-O netboot.tar.gz \
--quiet
$ mkdir -p /tmp/tftp
$ tar xf netboot.tar.gz -C /tmp/tftp

Install and Configure Dnsmasq

The dnsmasq is a convenient tool that brings together many network services, like DNS, DHCP, TFTP, and more. Dnsmasq is used here to setup a proxyDHCP and a TFTP server.

Ensure dnsmask is installed in the Vagrant box:

$ sudo apt-get install dnsmasq

Finally, create the file /etc/dnsmasq.d/proxydhcp.conf, configure the TFTP server and enable the proxyDHCP:

# Set the boot filename for netboot/PXE. You will only need
# this is you want to boot machines over the network and you will need
# a TFTP server; either dnsmasq's built in TFTP server or an
# external one. (See below for how to enable the TFTP server.)
dhcp-boot=pxelinux.0

# Loads pxelinux.0 from dnsmasq TFTP server.
pxe-service=x86PC, "Install Linux", pxelinux

# Set the root directory for files available via FTP.
tftp-root=/tmp/netboot

# Log lots of extra information about DHCP transactions.
log-dhcp

# This range(s) is for the public interface, where dnsmasq functions
# as a proxy DHCP server providing boot information but no IP leases.
# Any ip in the subnet will do, so you may just put your server NIC ip here.
dhcp-range=REPLACE_THIS_WITH_THE_IP_ADDRESS_OF_INTERFACE_USED_IN_THE_BRIDGE,proxy

Preseed the Installation

Preseeding is a method for automating the installation of the Debian operating system and its derivatives. Answers to installation questions, which would normally be answered interactively by an operator, are predetermined and supplied via a configuration file (and sometimes boot parameters). – Wikipedia

This process is sometimes called unattended installation.

In order to activate the unattended installation, append the preseed configuration to the boot command line in the installation menu (preseed/url=tftp://…/preseed.cfg) and activate the automation of the installation process (auto=true):

$ cat > /tmp/tftp/pxelinux.cfg/default << EOF
default install
label install
kernel ubuntu-installer/amd64/linux
append vga=788 initrd=ubuntu-installer/amd64/initrd.gz auto=true priority=critical preseed/url=tftp://REPLACE_THIS_WITH_THE_IP_ADDRESS_OF_INTERFACE_USED_IN_THE_BRIDGE/preseed.cfg
EOF

Finally, create a preseed.cfg file with the desired parameters for the installation, and place it under the TFTP root directory (in this case /tmp/tftp). Here‘s an example of preseed configuration.

Troubleshooting

How to check if the TFTP service is running?

$ netstat -tul

A TFTP service listening for connections should appear as follows:

udp 0 0 *:tftp *:*
udp6 0 0 [::]:tftp [::]:*

Check that I can get files from the TFTP server

Install the TFTP client and try to get files from the server.

$ sudo apt-get install tftp
$ sudo tftp
tftp: connect 192.168.1.106
tftp: get pxelinux.0
Received 43114 bytes in 0.0 seconds
tftp: quit

How to check if the proxyDHCP server is receiving the DHCPDISCOVERY packet with PXE-specific options?

$ sudo tcpdump -i enp0s8 udp -s0 -vvv

When the PXE client is started, UDP packets like the following one should be intercepted by tcpdump (note the NBP pxelinux.0 in the response):

21:03:00.491030 IP (tos 0x0, ttl 20, id 1, offset 0, flags [none], proto UDP (17), length 576)
0.0.0.0.bootpc - 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 00:26:b9:8d:de:4d (oui Unknown), length 548, xid 0xbb8dde4d, secs 8, Flags [Broadcast] (0x8000)
Client-Ethernet-Address 00:26:b9:8d:de:4d (oui Unknown)
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53, length 1: Discover
Parameter-Request Option 55, length 24:
Subnet-Mask, Time-Zone, Default-Gateway, IEN-Name-Server
Domain-Name-Server, RL, Hostname, BS
Domain-Name, SS, RP, EP
Vendor-Option, Server-ID, Vendor-Class, BF
Option 128, Option 129, Option 130, Option 131
Option 132, Option 133, Option 134, Option 135
MSZ Option 57, length 2: 1260
GUID Option 97, length 17: 0.68.69.76.76.49.0.16.87.128.90.195.192.79.81.52.74
ARCH Option 93, length 2: 0
NDI Option 94, length 3: 1.2.1
Vendor-Class Option 60, length 32: "PXEClient:Arch:00000:UNDI:002001"
END Option 255, length 0
PAD Option 0, length 0, occurs 212
21:03:00.492142 IP (tos 0xc0, ttl 64, id 52306, offset 0, flags [none], proto UDP (17), length 349)
192.168.1.113.bootps - 255.255.255.255.bootpc: [bad udp cksum 0xc373 - 0x374b!] BOOTP/DHCP, Reply, length 321, xid 0xbb8dde4d, secs 8, Flags [Broadcast] (0x8000)
Client-Ethernet-Address 00:26:b9:8d:de:4d (oui Unknown)
file "pxelinux.0"[|bootp]

Alternativelly, dnsmasq writes its logs in /var/log/syslog. Since we are using the log-dhcp option in the dnsmasq configuration, we should also have more information in that file.

My computer seems not to be booting in PXE mode

  • Does your NIC has to be configured through BIOS to have PXE enabled?
  • Are you using UEFI, and not BIOS?
  • Do you need to press some key during boot time to boot in PXE mode?

To answer all these questions you might need to check your computer’s manual.

Posted in Uncategorized | Tagged , , , , , , , , , | Leave a comment

Devoxx France 2016 : L’Odyssée du Continuous Delivery

Les 20, 21 et 22 avril a eu lieu à Paris la 5ème édition de la conférence Devoxx France. Issue de la communauté Java, Devoxx France s’est ouverte au fil des ans et accueille les développeurs en général.

Lors de cette édition, j’ai eu l’occasion de co-présenter un retour d’expérience sur la mise en place du Continuous Delivery à la Société Générale CIB (SGCIB), à laquelle j’ai participé en tant que coach technique. Depuis quelque temps, la SGCIB investit massivement sur le Continuous Delivery pour répondre à ses enjeux métiers.

Dans notre retour d’expérience sur un an de transformation vers le Continuous Delivery, nous parlons de notre passage en Feature Team, de l’adoption du Trunk-Based Development, de la mise en place des Release Train et du Toggle Feature, de Deployment Pipeline, de BDD, TDD, Clean Code, dette technique, entre autres.

Voulez-vous en savoir plus ? Clément Rochas a fait un retour sur notre présentation sur le blog de Xebia. Sinon, ci-dessous vous pouvez regarder notre présentation en vidéo. Les slides sont sur SlideShare.

Posted in Uncategorized | Tagged , , , , , , , , | Leave a comment

Soirée Software Craftsmanship au Paris JUG le 9 février 2016

Le 9 février 2016, j’ai eu l’occasion de co-animer avec Bruno Bouccard et Thomas Pierrain la soirée Software Craftsmanship au Paris Java User Group.

Cette soirée dédiée au refactoring du code legacy s’est déroulée en mode coding dojo. Le kata Trip Service a été utilisé pour l’occasion.

Durant l’exercice, Bruno a fait du live coding en C#. En face, les participants, majoritairement en Java, l’ont suivi en répétant les mouvements de refactoring.

Je tiens à remercier Bruno et Thomas pour l’invitation, à l’équipe du Paris JUG pour l’accueil et l’organisation et aux participants pour les échanges, questions, etc.

Pour en savoir plus sur les techniques utilisées pour remettre le code d’aplomb, une vidéo a été mise en ligne par l’équipe du Paris JUG.

Posted in Uncategorized | Tagged , , , , , | Leave a comment

Ruby: from ZERO to HERO

Gentle introduction to Ruby programming language.

This presentation brings an overview of basic concepts and useful information to start coding.

More useful links:

Posted in Uncategorized | Tagged , | Leave a comment

Extreme Carpaccio

At the end of 2014, I started working for a well known French investment bank as Software Craftsmanship evangelist. About 2 or 3 years before, an ambitious program started being deployed across the organisation in a huge movement towards Continuous Delivery. I joined the agile and technical coaches’ team in charge of the program as Craftsman Coach (as they use to call the role themselves). My job was to help development teams to improve software quality, stability and the delivery process, by introducing concepts like TDD, clean code, continuous refactoring, BDD, continuous integration, simple design, and so on.

On my day-to-day work, I strongly used coding exercises, better know as code katas, in order to help developers to improve their skills. Most of the exercises I used can be easily found on the internet. Nevertheless, sometimes I was led to create some exercises by my own to better address specific team needs. On that context, while trying to figure out how to better explain feature slicing and iterative development, Radwane Hassen (a colleague also working as tech coach) and I have created Extreme Carpaccio, which was inspired by two great existing exercises: Extreme Startup and Elephant Carpaccio.

Extreme Startup

Extreme Startup is a coding exercise created by Robert Chatley and Matt Wynne. The first time I played Extreme Startup was during a Xebia Knowledge Exchange (XKE) in a session organised by Sébastian Le Merdy, Nicolas Demengel and Jean-Laurent de Morlhon.

During an Extreme Startup session the facilitator uses his own computer to send HTTP requests with puzzles to participants’ computers. Participants should figure out how to solve each puzzle and code to respond server’s requests. For every good answer, the participant earn points and for wrong answers, he loses.

Extreme Carpaccio uses the same principle: a server simulates a market and sends requests to participants who need to code the response. It turns out this approach is very funny, since it creates a competitive atmosphere and people try their best to win. However, counter to Extreme Startup, in an Extreme Carpaccio session most of the problem is unveiled at the beginning and teams should decide what to implement first.

Elephant Carpaccio

Elephant Carpaccio is a slicing exercise created by Alistair Cockburn. Many colleagues use this exercise to work with teams and help them to improve their slicing skills.

In an Elephant Carpaccio session, the facilitator exposes the problem and asks participants to slice it into small pieces, prioritize and implement iteratively. At the end of each iteration, teams demonstrate how they devised the problem and what they achieved. The goal is to slice the whole problem into a bunch of small ones, in such a way that each sub-problem could bring feedback and value (either knowledge value, customer value, or both). The opposite approach (waterfall) would require a huge amount of time during the development phase and only allows delivering to production once everything is done. This approach has many drawbacks and one of them is that it postpones feedback.

Unlike Elephant Carpaccio, Extreme Carpaccio doesn’t guide participants through slicing, iterations and demonstrations. However, the exercise was designed to promote teams that slice the whole problem through a value perspective. The more you slice, the earlier you go to production, and the earlier you earn points. If you only go live when the whole problem is implemented (waterfall), people that took an approach based on iterations will start earning points before you.

Problem to be solved

Based on Elephant Carpaccio, the problem consists in a company which sends to each team’s computer (sellers) a purchase order. The order contains prices and quantities of each product being sold, the buyer’s country and the reduction (discount) to be applied. Using this information, teams need to figure out how to calculate the order’s total amount and send back to the server a bill containing the amount. If the answer is correct, the team earns points equivalent to the order’s amount. Otherwise, they lose half of the order’s amount. For each wrong answer, the server sends a feedback to the client with the right amount value.

Following an example of request the server sends to participants:

{
    "prices": [65.6,27.26,32.68],
    "quantities": [6,8,10],
    "country": "FR",
    "reduction":"STANDARD"
}

The number of purchase orders each country sends varies depending on the country who sent it. This number follows the population density of each country. Germany, UK and France, for instance, will send more requests than Luxembourg, Cyprus or Malta. That information is not mentioned explicitly to encourage people to exchange with the facilitator. The goal here is to favour communication and the facilitator(s) should encourage then teams to exchage with him/her/them. It is also possible to figure that out by looking at the messages sent by the server. Nevertheless, as in real life, asking questions to the right person can save you time.

Session Plan

Extreme Carpaccio was designed to be played with developers and product owners (or business analysts) together. Nevertheless, it is possible to run the exercice only with developers too.

During an Extreme Carpaccio session:

  1. participants get organised into teams of 2 or 3, with 1 product owner and at least 1 developer each (2 is better) – this should take around 5 minutes
  2. the facilitator exposes the problem and announces the next steps – takes generally between 5 and 10 minutes
  3. the facilitator asks teams to start slicing and prioritizing user stories – between 10 and 20 minutes
  4. the facilitator starts the server and asks teams to register themselves via the server’s dashboard – can take from 5 to 15 minutes, depending on the number of participants, network issues, etc.
  5. once everyone is registered, the facilitator allows people to start coding – takes between half and one hour
  6. (optional) if teams are fast and quickly solve the exercise, the facilitator can activate some new behaviour or traps in the servers which might shake the score and add more challenge
  7. retrospectives – around 15 minutes

Book 1h30 to 2h for the exercise. During the session, the facilitator should walk among teams in order to make sure everyone is on the run. Most of the problem is exposed at the beginning, but not all of it (see reductions’ section). Teams should try to decouple the problem based on their own strategy with the information they have.

At the end, take some minutes to do a retrospective and ask participants what they’ve learnt and if they have something to share. I generally ask people who finished at the top which slicing strategy they chose. You can also ask about technical difficulties, success, etc. Retrospectives are frequently very rich and I’m frequently surprised to discover new ways to deal with the problem.

Extreme Startup session at Software Craftsmanship Paris:

This slideshow requires JavaScript.

Changing Behaviour And Adding Constraints On The Fly

In the middle of a session, generally things start to stabilize and we will probably have some teams at the top and the others in the middle and down in the ranking. At this moment, in order to challenge teams on top and shake the score, the facilitator can change the order’s reduction type without saying anything. Since the reduction rules used to calculate the new amount aren’t described anywhere, people need to stop doing what they are doing and try to figure out how the new reduction works (this can be easily achieved by looking at the new purchasing orders and feedback coming from the server). By doing this, we allow people which were in the mid-raking to challenge people on the top. This breaks with the “predictable” aspect of the exercise, that happens when some teams perform much better than the others from the beginning.

Beside the reduction, other behaviour/constraints are available to the facilitator if he/she/them want to add more challenge during the session or shake the scoreboard. It is possible, for instance, to start charging teams with penalties when they go offline. This requires teams to think “zero-downtime” and keep an eye on production (contribution from Julien Jakubowski). It is also possible to send corrupted orders, making teams’ servers crash. This tends to encourage testing, mainly in boundaries. Finally, the facilitator can use a JavaScript function to overrite the formula used to compute taxes for individual countries, making taxes computation more dynamic (and breakes the dictionnary pattern most teams use to keep countries’ taxes) (contributions from Arnauld Loyer).

Interesting Things I’ve Faced Running Extreme Carpaccio

Something interesting I’ve noticed running Extreme Carpaccio sessions is how teams slice when someone is playing the role of product owner and when there are only developers in the audience. Generally, developers tend to release once the whole implementation is ready. Product Owners usually gather developers around a slicing strategy and prioritize user stories in order to earn money (points) as soon as possible.

However, an interesting fact that sometimes happens when there are mainly developers in the audience is that they try to hack the server as well as other teams. Personally, I find this great because it pushes people to think out of the box. Generally, participants try to hack their own friends and someone they don’t know only when we’ve established together from the beginning that “everything is possible”. Some hacks I’ve faced were DDoS against the server, man in the middle or someone sending fake requests trying to usurp server’s identity.

Extreme Carpaccio sources are available on Github. If you want to contribute, create an issue on the Github or make a pull request. If you run an Extreme Carpaccio session share your feedback on a blog post or on Twitter and let me know.

Updated on 11/12/2016: add changing behaviour/constraints on the fly.

Posted in Agile | Tagged , , , | 1 Comment

Java Performance: what developers must know

I have observed that developers (myself included) tend to focus more on the make it work and make it right stages of building software, and less on the make it fast part. The consequence? Sometimes our applications behave very badly under significant or even moderate load, and we generally realize that too late… 🔥

Recently, I’ve been involved in solving performance issues at many levels, mostly on Java applications. In this presentation, I list some tools and techniques that were very helpful to me in order to find, fix and prevent from performance issues.

Posted in Java | Tagged , | 2 Comments