How satellite and aerial photography brings the war diaries to life
by Cynthia Gast
Most people, and especially the volunteers working on Operation War Diary, will recognize the names of the locations that played the biggest roles in World War I. Cities such as Ypres and Passchendaele along with regions such as the Somme were the sites of horrific fighting. Poperinghe was known to the soldiers as a place where they could have a break behind the lines, and the unfortunate mademoiselle from Armentieres is immortalized in song.
There are countless other towns, villages, farms, and forests that appear in the diaries. Each one is a place where the men of that unit fought, marched, or slept. To me, an important part of the story of the men on the Western Front is where they were. Where did they spend Christmas? What did they see in the villages behind the lines where they spent their rest periods? Where was that hill that caused so much trouble for the transport on a dark rainy night? I’ve spent a lot of hours searching for some of these locations.
The first step, even before filling in the Place tag, is to search for the name on Google Maps. I’ve learned that not all diary authors had a good grasp of the intricacies of spelling French place names and the search function in Google Maps helps with this problem by supplying several suggestions before I’ve even finished typing in the entire name.
Google Maps has a couple of other helpful features as well. I’ve learned that clicking on the town’s name on the map puts that name in the tab at the top of my browser. This is quite handy for spelling when I return to the diary page to type it into the Place tag. Another interesting detail is that while many of the very tiny hamlets are not labeled on the basic Google map, they do show up on the Google Earth view. An example of this that I came across recently involved the 1 st Division Trench Mortar Batteries. Their diary said they were in the village of Berthaucourt and I knew they were somewhere near Saint-Quentin. A search on Google Maps told me that Berthaucourt is now part of Pontru, but no matter how closely I zoomed in, there was no indication of its exact location. However, when I switched the view to Google Earth, I learned that it is just to the east of the crossroads that form the heart of Pontru.
As I’ve moved through the diaries, I’m less content with locating just the villages and towns mentioned by the author. These days, I’m digging deeper and looking for the farms and chateaus used as billets and bivouacs. I’m amazed at how many of the farms mentioned in the war diaries are still in existence today.
When I want to find a farm, I start on Google Maps, where search results are more likely to show me a road rather than the farm itself. However, when I zoom in on Google Earth view there’s often a cluster of farm buildings along that road. Sometimes I hunt using Google’s basic search function, which is particularly useful for chateaus.
Many World War I trench maps are now available online. Some sites offer sets for purchase but, along with IWM’s map collection, there are at least two sites that allow users to examine them at no charge. The National Library of Scotland and McMaster University in Canada both offer collections covering the areas where most of the fighting occurred. They both also offer information and links that cover how to read the maps, including the grid references. As an added plus, McMaster University also has a collection of aerial photographs of the Western Front.
Trench maps are fascinating. They reveal the complexity of the trench systems and show just how the features of the landscape fit into the systems. They also show the sometimes dark sense of humor possessed by the men who named the trenches. Whole sections of trenches are named for weapons, animals, or birds, as well as places back home, commanding officers, and the units occupying them. One system near Delville Wood featured trench names relating to beer, such as Pint Trench, Hop Alley, Ale Alley, Beer Trench, and Lager Lane.
Trench maps also often include farms, chateaus, villages, and anything else of importance to the men in that area, such as quarries and brickstacks. They also indicate the roads existing at the time the maps were drawn, many of which still exist today. When I want to find a farm or chateau, I compare the trench map with the modern map or Google Earth view using the roads as a guide. The National Library of Scotland also offers side-by-side and overlay features to help compare the old and present-day maps.
When I began as a volunteer at OWD, I had no idea of the things I would learn. The geography of the Western Front and how to read trench maps are just two of the amazing aspects of World War I that I now know. There are still many diaries to read and men to get to know through the official diaries they kept, and lots of spaces for more volunteers.
Operation War Diary has been running for over two years now. Together, we have placed hundreds of thousands of tags, made similar numbers of comments, and followed the journeys of hundreds of units through the conflict at the Western Front.
And, like all things, we have evolved over that time.
When we began, we followed in the footsteps of other great crowd-sourced digital humanities projects like Old Weather. But the content we are dealing with at Operation War Diary is unique in its depth, breadth and richness. It meant we had to make certain assumptions when we started out.
Mainly, this was around what should and shouldn’t be tagged, which in turn was based on what we thought the data we would produce might look like and how it would be used. In part, we were led by the transcription mantra, which is that only what is there should be written down. However, tagging is a very different activity to transcription, with a quite different set of applications.
Under our initial guidance, volunteers tagged only what was explicitly mentioned on a diary page, and we also told them not to tag certain everyday activities for units like ammunition columns, mobile veterinary sections and engineers – the movements, collections and checking of infrastructure which might be considered the bread and butter of the units in question.
In part, this was to make the process less onerous for our taggers. We have 1.5 million pages to get through, after all! But, as I said before, it was also partly because we hadn’t quite left the transcription mindset behind.
However, we now have our first real use of Operation War Diary data to refer to, courtesy of Professor Richard Grayson, and it makes for very interesting reading. If you haven’t read the article already, you can find it here.
To some extent, the quality and richness of the data which can be used to support studies like this is limited by what was included in a diary in the first place – some are much sparser than others. However, by following the transcription-oriented method of only tagging what we can see, are we also unnecessarily reducing the coverage of the data we produce?
What about the case of a unit which we know to be in the line, because the author tells us so on one day, but over the course of the next four or five day’s worth of entries, that fact isn’t explicitly mentioned again? Very often, it’s clear that the unit is still in the line, but that information is then lost because there’s nothing for us to drop a tag on.
Or the Mobile Veterinary Section who spend a week travelling from place to place, picking up sick horses to take back to the depot? Again, under our starting assumptions, that detail would also have been lost, because we felt it wasn’t necessary to tag activities we already knew certain units spent much of their time doing.
That’s fine from the standpoint of our knowledge and common-sense understanding of these units and the functions they carried out during the war. But if we shift the perspective to one of providing evidence, quantitative facts which we can use to illustrate our understanding, then by not tagging certain things we know to be true, we aren’t realising the full potential of Operation War Diary.
Of course, there’s a line between inferring what to fill the blanks with and making things up, but as our understanding of the project evolves, so too does the knowledge and experience of our long-term taggers, who may have started off knowing very little about the war diaries, but who have now read and tagged hundreds, if not thousands of pages and are very well placed to see patterns in the information and extrapolate from what is written down to what is only implied.
That will mean making judgement calls at times, but the Talk forums provide a great environment for testing out any inferences before we press the ‘Finish’ button. The whole concept of Operation War Diary is that it is built on consensus, so why not extend that to these situations too?
There are practical issues to overcome – where to place a tag for an inferred activity, for example, or which tag to use. For the former, I would suggest dropping inferred tags close to the date to which they should be linked – our clustering algorithm will then group them together and ensure the information is recorded in the way it was intended. For the latter, we may have to recourse more frequently to the unsatisfactory ‘Other’ option for activities which do not fit neatly into the standard list, but that at least will still allow us to build up a comprehensive timeline for each unit and will clearly indicate what they were not doing, even if we can’t provide specifics beyond that.
With our first published use of Operation War Diary’s data, I believe we now have a clear and compelling case for tagging as much information as we can as accurately as we can. And that is the beauty of Operation War Diary – we can evolve and improve what we do and, in doing so, can tell the stories of the Western Front in the most effective way we know how.
On the 26th of April, 1915, the 15th Ludhiana Sikhs were assembling ready to move up into the Ypres Salient. Shortly after they moved off, the march was halted due to enemy shelling. All non-essential transport was left behind and the Sikhs continued onward by companies, leaving a three minute gap between each. Around them was absolute chaos – heavy traffic in both directions, transports full of wounded, groups of refugees, other bodies of infantry. They pressed on.
‘The spires and towers of Ypres soon came in sight and simultaneously the whistling and screaming of shells followed by loud explosions as they fell into the unhappy town.’
Imagine the sight – endless salvos of shells slamming into this ancient and beautiful town, history being undone right before their eyes.
The Sikhs left the main road then, passing north of the town, where comparatively few shells were falling. The diary author notes that it was still sufficiently unpleasant to keep them from dawdling on the way.
It was here they came under fire from 42cm shells for the first time. Probably fired from a Big Bertha howitzer (a super-heavy mortar) these enormous shells tore through the air with a noise…
‘…best compared to that of an express train’.
Enormous columns of debris and smoke were thrown 300ft into the air. You can almost sense the diary author’s awestruck horror and, just for a second, imagine the hellish noise and destruction they must have been marching through. Have a look for yourself; the links to the diary pages are below:
This episode is just one of many where the dry, official tone supposed to be adopted by diary authors gives way to something much more personal. It’s difficult for us to imagine exactly the things these men experienced, but as their personalities and feelings start to find their way into their words, it’s possible to feel a connection to a human being, rather than an historical account. One hundred years might separate us from them, but it no longer seems like quite such a long time. Josie Pegg, the Citizen Historian who first came across this account finds it…
‘…very moving to read accounts that reveal poignant details and the emotions of the men involved. I originally started tagging the diaries as an interest when I retired and was lucky to be able to start with the 11th Leicester Pioneer battalion as my Great Grandfather served with them. The many small incidents and asides that are mentioned in most of the diaries have been fascinating. It has also become apparent that life for the average soldier was mostly mud, boredom and waiting around, interspersed with moments of horror.’
Why not join Josie and the rest of our Citizen Historians in preserving these links to the men who chronicled the First World War? With your help, their legacy will live on for many generations to come. Get involved at Operation War Diary.
Recently, Steve Hirschorn at the National Archives has been looking at ways to visualise the information being generated by Operation War Diary Citizen Historians. As part of this work, he has taken maps found in the pages of the war diaries and, using the known coordinates of certain features contained within them, has fitted them to current satellite images using Google Earth. This process is known as georectifying and can help us assess how much the landscapes described in the war diaries have changed in the 100 years since they were written.
Over to Steve, who can tell you exactly how he’s done this…
Thanks to the efforts of volunteers using the #map hash-tag in the talk forum, it’s been easy for me to find georeference-able content. By geo-rectifying maps, it would be possible to use a GPS device to find the exact co-ordinates of anything that is documented in the maps, such as trench locations and routes, and machine gun emplacements.
Four KMZ (Google Earth/Google Maps) files are linked below. If you have Google Earth (free download available), you can use these files to view First World War maps overlaid on a recent satellite image. The KMZ files can also be imported into Google Maps, but the functionality via the website is more limited compared to the full Google Earth client.
After you have downloaded a KMZ file, you’ll see the war diary maps overlaid on the satellite view, but completely opaque. Look in the left-hand menu on Google Earth and by right-clicking an individual map item, you can select Properties and in the properties dialog box that appears, you can adjust the transparency from 0% to 100% and anywhere in between. I’ve tried to identify evidence in the current-day satellite images and street view photos of trench locations, but haven’t had much luck so far.
I’m amazed how well the most of the maps fit the current day landscape features. The odd road has disappeared here and there, but there are always enough reference points to fit the map, and they usually fit with just a bit of stretching and rotating, all of which functionality is also available in the free Google Earth client. A couple of them show trench locations (the map of Bullecourt) and machine gun placements (Ypres).
If you missed the link the first time, Google Earth can be downloaded here: http://www.google.co.uk/intl/en_uk/earth/
Steve’s map files can be downloaded using the following links:
- http://zooniverse-demo.s3.amazonaws.com/diaries_data/images/WO-95-1662-1_Bullecourt.kmz One of the better ones, a good fit to current-day road layout and showing the locations of trenches
- http://zooniverse-demo.s3.amazonaws.com/diaries_data/images/WO-95-1601-2_Ypres.kmz Another good fit, this time showing machine gun emplacements
- http://zooniverse-demo.s3.amazonaws.com/diaries_data/images/WO-95-1415-1_Hooge.kmz A map showing German trenches and the British Front Line
Lastly, as this is a new area of exploration for us here at Operation War Diary, and because none of us are experts on it, we have some questions which we hope you might be able to help us with:
- Is KMZ the best, most open format for sharing geo-rectified maps? Is there a better format?
- A bit of researching on the Internet suggests that there are ways of loading KMZ files onto a SatNav. Again, are KMZ files the best way of supporting this?
- Are there any web-based applications that enable geo-rectification of maps, and also provide a method of sharing geo-rectified maps?
- Do you have any geo-rectified maps you’ve created that you’d like to share?
- Are there any other ideas for things that we can do with the maps?
Post your answers in the comments here, or get involved on our forums at: http://talk.operationwardiary.org/#/boards
Today’s guest blogger is Heather Collins, a volunteer moderator on Operation War Diary and also one of the Citizen Historians involved in tagging the diaries. She first became interested in WW1 through researching her own family’s contribution to the conflict and has been fascinated by it ever since.
The 2011 film War Horse brought the role of horses in 20th Century warfare back into the public focus and reminded us how much we relied upon them in an age where motor transport was not yet able to go everywhere. On the Western Front in World War 1, horses were used for moving all types of supplies and equipment as well as providing the fast moving fighting arm of the Cavalry. But even knowing how important the horse was, some of the figures relating to their use are still staggering to read:
- Over 1 million horses were sent overseas to war from the USA
- By 1917, 1000 horses per day were arriving in France as remounts for the British Army
- Casualties amounted to about 1 horse lost for every 2 men throughout the war
- Horse fodder was the largest single commodity shipped to the front by Britain – with a higher tonnage even than ammunition!
At the start of Operation War Diary, there was no way to track information about horses at all and pretty much from the first few days of tagging, users were asking why and wanting to be able to follow the story of the horses as well as that of the men. Early on we suggested that people use the hashtag #horses but it soon became obvious that as horses get a mention in just about every page of some diaries, this wasn’t really going to bring out much useful information! A rethink was needed if we wanted to make best use of the obvious desire of Citizen Historians to follow the fate of the horses.
Following some discussion on the project’s talk forum pages, which shows one of the strengths of a crowd-sourcing project like OWD, where users can get involved and have a real influence, we decided to try to define some more specific hashtags to follow particular aspects of the life of the horse on the Western Front.
Two months later we now have hashtags telling us about the condition of the horses, their food, their ailments, horse casualties and their evacuation into the veterinary chain. We have also developed hashtags to look at the condition of the roads and the problems with water supplies as a direct result of these initial ideas.
We already knew how important the horse was in the appalling front line conditions in France and Belgium. Even though motor transport existed, it was often completely incapable of even moving in the quagmire that developed close to the front lines and horses were sometimes literally the only way of getting the heavy equipment and supplies forward. For example in 1914 a single battery of the Royal Horse Artillery with 6 x 13 pounder field guns required over 200 horses to provide the motive power to move men and equipment quickly. So having these additional hashtags has enabled us to look at everyday details that the Units were recording in their War Diaries regarding their horses.
Here are a few snippets of the kind of information we have found:
Water: Units seem to have been constantly struggling to find an adequate water supply for their horses. In September 1916, the 2nd Divisional Ammunition Column sent 1000 horses to the rear area due to water difficulties.
Food : An Artillery battery describes the ration being cut in February 1917 to 9 pounds of oats and 10 pounds of hay as a result of which “horses lost condition during the cold owing to short food”. This was apparently due to increased German U-Boat activity in the Atlantic having reduced the amount of oats coming in from the USA.
Equipment: An ASC Company in the 2nd Divisional Supply Train talks about having “rough shod” their horses in the snowy weather. This is a process whereby shoes with protruding spikes or nails are fitted to enhance grip and it is the origin of the phrase “to ride roughshod over”, The diary goes on to describe the difficulty of fitting the attachments, which they call “cogs”, to the shoes where the existing shoes were already worn thin.
Fatigue: A Cavalry unit in August 1914 describes how the long marches (27 miles on this particular day) are taking their toll on the horses “Some were under saddle 20 hours and water was scarce. Most would neither eat nor drink when they got into bivouac” Most cavalry units already had a number of dismounted men due to the strain on the horses and the Veterinary Corps units were struggling to keep up with the numbers, many of whom simply required rest and food.
Casualties: Mobile Veterinary Sections (MVS) often talk about “horse floats”. This turns out not to be some sort of boat, but actually to be another name for a horse trailer or horsebox. These were being developed so horses that might otherwise have to be shot or abandoned could be recovered to their unit or to the MVS. In February 1915, 7 MVS report having visited a horse left by its Unit on a farm and they suggest it might be collected in a week’s time by another MVS “who have the use of a horse float”. The tone of the diary makes it clear the writer thinks all MVS Units should have such a modern innovation!
Why not join us at Operation War Diary and help us to find out more?
Tens of thousands of maps were produced by British forces during the first world war, from large-scale maps for senior commanders needing an overview of an entire front, to much more detailed topographical maps for front-line troops, which allowed them to accurately pinpoint enemy positions and establish the nature of the ground they held.
Many of these maps are preserved in the unit war diaries, with Citizen Historians uncovering more each day. One of the great benefits of tagging the war diaries is that it allows us to continue the mapping work carried out during the war itself, using modern techniques to establish a visual representation of the events of the time and the experiences of the units to whom the diaries belong
Image © IWM (Q 2306)
One way in which we can do this is to use place names, particularly those we can locate on the mapping tool contained within the Place tag, to site events geographically, which is a great way to quickly get an overview of a particular type of activity or event. The link below, for example, will take you to a map produced by Jim O’Donnell of the Zooniverse showing amalgamated unit casualty figures (tags showing 0 casualties have been discarded for this analysis), linked to geographic location and covering the years 1914-1918.
Bear in mind that this data set is derived from the first tranche of completed diaries – a little over 200 units in all. Imagine what the map will look like once we have completed all of the thousands still to be tagged!
Georectification is another exciting way in which mapping tools can add to our understanding of the content of the war diaries. This technique allows us to align first world war maps discovered in the diaries with modern satellite images of the same area. We’ll be doing another blog post soon to cover this and you’ll get to see some excellent examples of just what we can do – give yourself a head start by downloading Google Earth beforehand!
With all our Citizen Historians gallantly tagging away, we thought it was high time we explained how all that hard work is being used to produce the data sets for the project.
While we really appreciate all the effort each and every individual is putting in on the diaries we know that errors can arise for one reason or another. For that reason, we generate what’s known as consensus data. We have an algorithm that allows us to do this.
To begin with, each of the diary pages is tagged by at least five Citizen Historians. Five different people who might each look at that page in a slightly different way. Once that tagging is complete, the diary page is closed and put in the queue for processing.
The system starts this by identifying tags of the same type relating to the same entity (a place, a person, an action etc.). It has to take a best guess at this, clustering tags together based on a percentage of the image size for each scanned diary page. Trial and error has shown that this percentage is best set at 3% vertical and 10% horizontal. There must be a minimum of two tags for a particular entity if it is to make it into the final consensus data set. So, if two of the five Citizen Historians who have tagged a diary page have both identified a place in the same position on that page, that place makes it in.
Image © IWM (Q 5700)
The consensus tag generated from this tag cluster is then placed at the average location in which all of its constituent tags were generated
Next the system has to determine exactly what information should be attached to each tag. This is relatively straightforward when the original tags came from a fixed list (e.g. Activities tags, which can be of only a certain number of types). Where tags contain free text (e.g. person or place), fuzzy text matching is used to determine their attached information (e.g. Slater-Booth, Sclater-Booth and similar variants would be grouped together). Where a majority of these free text tags have the same value, that value becomes the consensus value. However, if there is no clear majority value, then the consensus tag will be formed of the leading variants.
The algorithm is also designed to create serialised data. In essence, this means that each consensus tag is associated with a date, which allows the data generated to then be ordered by date. When Citizen Historians tag dates on a diary page, they essentially segment that page, and it’s that segmentation which allows the system to determine which consensus tags should lie inside which date area.
Once these operations have been carried out for one page of a diary, the next page will be processed and so on until the diary is complete.
Don’t worry about us losing all the tags you’ve generated, though – our databases hold everything that every single one of our Citizen Historians has added to Operation War Diary, be it individual tags, hashtags or text comments. We know just how valuable a resource that’s going to be for anybody wanting to investigate the diaries beyond the standard, structured tags we’ve defined.
Why not check out our first batch of consensus data here: http://wd3.herokuapp.com/public
As part of an ongoing blog feature, researchers involved with Operation War Diary will be introducing themselves and discussing what the project means to them and their work. This week Chris Kempshall, Project Officer for East Sussex County Council’s WW1 Commemorations preparations and Visiting Researcher and Associate Tutor at the University of Sussex, talks about his research interests and how Operation War Diary will help him develop them further.
Over to you, Chris!
I have had a circuitous approach to the field of First World War studies with my initial university BA degree being in Media Practice and Theory, followed by a Masters in Contemporary War and Peace Studies. It was when I encountered theories of the ‘myth’ of the First World War during history related courses in my undergraduate that the war first took hold as an interest of mine and Dan Todman’s work on the subject cemented my interest in the field. At the time, rather naively, I found the idea that history could somehow lie or lack truth to be extremely unfair.
From this starting point I then, over the course of a few years, ended up becoming more intrigued with the nature and construction of the First World War. The original notion behind my PhD thesis was to examine whether the British and French had competing or complementary ‘myths’ of the First World War but it was during that early research that I realized that the relationships between British and French soldiers had been largely overlooked and the more I examined it the more interested I became. The article Elizabeth Greenhalgh wrote on ‘Parade Ground Soldiers’ proved to my mind how important an area this could be.
As a result my research now focuses predominantly on the interactions and relations between allied soldiers of different nationalities. My PhD thesis moved on to focus on the relations between British and French soldiers on the Western Front. Whilst previous studies have touched on the relations between common soldiers, this has often been within specific case studies. I drew particularly on the contemporary diaries, letters and written records of British soldiers within the Imperial War Museum and also the postal censorship records of the French army at the Archives de l’armee de terre in order to trace the nature and evolution of these relations across the war.
One of the main areas of interest for me with Operation War Diary has been examining the records which report on interactions between different allies. Having just recently been appointed to the Academic Advisory Board for the Imperial War Museum I am extremely excited at seeing the results of all this public research.
Alongside my research I also teach and lecture on a First World War course at the University of Sussex and, for the next eight months, am the Project Officer for East Sussex County Council’s ongoing project of World War One Commemorations. I am also preparing my PhD thesis for possible future publication.
You can make a real difference to Chris’ work by using the hashtag #alliedrelations if you come across anything you think might be relevant while tagging the diaries. Relationships don’t need to be limited to those between the British and French forces!
In the first eight weeks since the launch of Operation War Diary, over 10,000 citizen historians worldwide have tagged names, places and other details in over 200 unit war diaries.
Initial reports reveal some amazing statistics:
- Over 260,000 tags relating to named individuals
- Over 332,000 tags relating to places
- Almost 300,000 tags relating to activities
- The amount of volunteer effort put in so far is equivalent to one person working 40 hours a week for four years.
With your help we’re going one step further than traditional transcription by using the data to digitally map and analyse patterns and trends in the unit war diaries, offering new perspectives on the First World War. Our developers and academic advisory group are hard at work crunching the numbers from the first two months of the project – we’ll blog about their plans soon.
In addition, much of the data – particularly names and places – will be integrated into The National Archives’ catalogue (Discovery), allowing researchers and family historians to search the diaries for named individuals mentioned in the diaries. Making the data freely available to researchers in this way is hugely important to all of the project partners, and we want everyone to be able to benefit from the amazing efforts put in by citizen historians. The data will be published in this way under the Open Government Licence.
The data will also be available to users of Lives of the First World War, which we’ll tell you more about soon.
Find out more about the unit war diaries
Today The National Archives has published another 3,987 digitised First World War unit war diaries from France and Flanders online, which means that around 6,000 diaries are now available on The National Archives’ website to search free of charge (and download for a small fee). These will in time be added to Operation War Diary for tagging.
If you’d never encountered a unit war diary before tagging them for Operation War Diary, you may have been wondering what all the fuss is about and why they’re so important to researchers and historians. They contain a wealth of information of far greater interest than the army could ever have predicted, providing insight into daily events on the front line, and are full of fascinating detail about the decisions that were made and the activities that resulted from them.
If tagging the unit war diaries has inspired you to find out more, whether about a particular unit or an individual, The National Archives has many useful online resources that should help you. Our First World War 100 portal gives an overview of the millions of records that we hold relating to the war, from war diaries to conscription appeals via service records and Cabinet papers. It’s an essential starting point for anyone researching a First World War ancestor, with step-by-step guidance to help you on your way. We also have a wealth of multimedia resources, including podcasts and the popular My Tommy’s War blog series. We send out a free monthly enewsletter with news and updates relating to our work and collection – sign up today to receive your own copy!
The National Archives