Announcements
The new Eurosurveillance website is almost here.

Eurosurveillance is on the updated list of the Directory of Open Access Journals and in the SHERPA/RoMEO database. Read more here.

On 6 June 2017, the World Health Organization (WHO) published updates to its ‘Essential Medicines List’ (EML). Read more here.

In this issue

 Home Eurosurveillance Edition  2017: Volume 22/ Issue 32 Article 3
 Tweet

Eurosurveillance, Volume 22, Issue 32, 10 August 2017
Research article
Pelat, Bonmarin, Ruello, Fouillet, Caserio-Schönemann, Levy-Bruhl, Le Strat, and the Regional Influenza study group: Improving regional influenza surveillance through a combination of automated outbreak detection methods: the 2015/16 season in France

+ Author affiliations

Citation style for this article: Pelat C, Bonmarin I, Ruello M, Fouillet A, Caserio-Schönemann C, Levy-Bruhl D, Le Strat Y, the Regional Influenza study group. Improving regional influenza surveillance through a combination of automated outbreak detection methods: the 2015/16 season in France. Euro Surveill. 2017;22(32):pii=30593. DOI: http://dx.doi.org/10.2807/1560-7917.ES.2017.22.32.30593

Received:24 October 2016; Accepted:19 April 2017

## Introduction

The influenza A(H3N2) strain that circulated in France during the winter 2014/15 caused an overload of patients in hospitals’ emergency departments (ED), with more than 30,000 visits, and in intensive care units, with at least 1,600 severe cases [1]. The hospitalisation rate after ED visit was high (11% vs 6–9% usually since 2009), especially among elderly people (47%).

Santé publique France, the French national public health agency, announced the start of the epidemic at the national level when the national incidence rate of influenza-like illness (ILI) cases who consulted general practitioners belonging to a surveillance sentinel network crossed the epidemic threshold (determined using a periodic regression model) [2,3]. However, this was too late for many hospitals to prepare.

There was no communication about the epidemic’s imminent arrival in different administrative regions of France, although it is known that the start of the epidemic is not the same in all regions. Weekly influenza reports were produced in the national public health agency’s regional units, based on a variety of data and statistical methods, with no inter-regional comparison.

The national public health agency’s influenza alert procedure was subsequently reorganised to improve timeliness of alerts by implementing the following surveillance system features: (i) A pre-epidemic alert level was introduced to better anticipate the rise in ED visits and bed occupancy rates needed during epidemics; (ii) Epidemiologists in the national public health agency’s regional units were delegated the responsibility of determining the influenza alert level in the 22 administrative regions of metropolitan France (the part of France in Europe) every week and informing regional health authorities of the arrival of the epidemic in their region; (iii) The determination of the alert levels was aided by the standardised use of data sources and statistical outbreak detection methods.

In order to implement these three measures and help epidemiologists make better informed decisions about influenza alert levels, we developed a web application called MASS (Module for the Analysis of SurSaUD and Sentinelles’ data). MASS compiles data from two surveillance systems in near real-time, computes statistical alarm levels and proposes data visualisations that highlight inter-regional comparisons. Although influenza was our primary target, our goal was to create a tool suitable for the monitoring of various health indicators, in particular seasonal infections. Accordingly, 11 other so-called ‘syndromic groups’ were also included in this project: five with (most probable) infectious origin and winter peaks (acute lower respiratory infections, bronchitis, bronchiolitis, gastroenteritis and pulmonary diseases), asthma, isolated fever, meningitis, faintness, food-poisoning and wild mushroom poisoning.

In this paper, we present MASS, focusing on influenza. We describe its role within Santé publique France’s new surveillance workflow during the 2015/16 influenza season in metropolitan France and discuss its added value.

## Methods

### Data sources

SurSaUD: Emergency departments and emergency general practitioners

In 2004, Santé publique France set up the syndromic surveillance system SurSaUD, based on data from ED attendance, all-cause mortality and, from 2006 onwards, also data from emergency general practitioner (GP) visits [4,5]. Below we describe the SurSaUD data used in MASS. Mortality data were not used in MASS since these data usually lag behind morbidity data and therefore are not helpful in detecting influenza epidemic periods. Also, only the data after 1 January 2010 were used in MASS, as this corresponded to the time when the system reached a sufficient and stable coverage of the French population.

ED data are collected from computerised medical records completed during consultations in the hospitals involved in the Organisation of the coordinated surveillance of ED network (organisation de la surveillance coordonnée des urgences (OSCOUR)) [6]. This network grew from 23 hospitals in 2004 to 650 in 2016, capturing ca 88% of all ED visits and covering all French metropolitan administrative regions.

The participating hospitals transmit data to Santé publique France every day before 06:30. The files sent on a given day contain data on the patients seen during the previous 7 days. Most of the patients’ information is transmitted the day following the visit.

The transmitted data contain among other information: the diagnosis code according to the 10th international classification of disease (ICD-10), the consultation date and the patient’s age.

SOS Médecins is a federation of private emergency GP associations. In 2006 it started participating in SurSaUD with 24 associations, reaching 61 in 2016 (out of a total of 62, i.e. covering ca 95% of emergency GP consultations). All 22 metropolitan French regions have at least one participating association. When a patient calls an association to organise a home visit, their age and primary health complaint are registered. After consultation, the doctor registers the diagnosis code on a mobile application, using a specific thesaurus. Every morning, participating associations send their data to a national platform which synthesises all data into a single file which is sent to our agency at 06:00. The transmitted information contains among other details: the SOS Médecins diagnosis code, the consultation date and the patient’s age.

The received data are inserted in a database. An aggregated table is automatically updated with the daily number of visits by reporting unit (hospital ED and GP association), age group (several partitions) and syndromic group. More specifically, for ED data, the syndromic group ILI is composed of ICD-10 codes J09, J10 and J11. For SOS Médecins data, it comprises the three following codes from their own thesaurus: ‘ILI or suspected influenza’, ‘influenza confirmed by test’ and ‘pandemic influenza confirmed by test’.

Every morning, MASS extracts from this aggregated table the ILI data of the last 7 days corresponding to the following age groups: all ages, < 2, < 5, < 15, 2–14, 5–14, ≥ 15, 15–64, 15–74, ≥ 65, 65–74, ≥ 75, ≥ 85 years. MASS calculates, for ED and SOS Médecins data, the proportion of ILI cases among the total number of visits with a valid diagnostic code, which comprise all diagnostic types: diseases, accidents, etc. The proportion of ILI is calculated by age group, geographical area (France, metropolitan France and administrative regions), and by day, week and month. This indicator is preferred over the raw number of ILI cases, since the latter is sensitive to changes in the number of reporting units and changes in the proportion of coded diagnoses.

Sentinel general practitioners

Sentinelles’ is an electronic surveillance system set up in 1984, which collects individual data about consultations in general practice in the general population, using a network of ca 455 volunteer sentinel GPs (in 2015). They represent 0.76% of all GPs in metropolitan France, with wide variations across administrative regions, (from 0.3% in Picardy to 7.1% in Corsica) [7]. The definition of ILI used by Sentinelles is: sudden onset of fever > 39°C (> 102°F) with respiratory signs and myalgia. Sentinelles estimates the weekly incidence rates of ILI consultation in general practice for 100,000 inhabitants using a Horvitz–Thompson estimator in which the sampling weights of the ILI cases depend on the number of GPs participating in the surveillance, the total number of GPs, and the number of inhabitants in each area of interest, each week [8]. This implies that incidence rate estimates can be compared in space and time despite differences in GPs’ participation, overall numbers of GPs, and population size. However, the confidence interval of each weekly estimate depends on the number of participating GPs in the area in that week.

Sentinelles has developed an R package to make the estimates of all-age incidence rates and their confidence intervals accessible to external applications such as MASS. Until February 2017, when a new version of the package was released, the incidence rates by age group were not accessible through this package, and thus were not included in MASS.

The Sentinelles ILI data included in MASS are the all-age incidence rates of ILI consultation in general practice and their confidence intervals, from week 2010-W01 onwards, for the whole of metropolitan France and for all administrative regions. Every Monday at 14:00, Sentinelles releases preliminary estimates of incidence rates for the previous week. Every Tuesday at 10:30, Sentinelles updates these estimates, based on the data reported on Monday. Besides, because the sentinel GPs can report case data up to 12 days after consultation, the incidence rate estimates are updated during the 2 weeks following their initial release. Consequently, to provide up-to-date data in MASS, the all-age incidence rates for the 3 preceding weeks and their confidence intervals were downloaded from the Sentinelles database every Monday at 14:00 and every Tuesday at 10:30.

Comparison of case characteristics

Common case information reported by all data providers included age, sex and whether the cases were hospitalised following their visit to the ED or encouraged to do register at a hospital by its Sentinelles or SOS Médecins GP. These pieces of information were not available in real time from Sentinelles and were requested retrospectively at the end of the 2015/16 season.

We compared the case distribution by age groups, sex and hospitalisation status to understand the specificities of each data source, as this knowledge could be useful for future refinement of MASS. The 95% confidence interval of the proportion of cases in each group was calculated using 1 − (ps × qs × rs,v) as a finite population correction factor, ps being the coverage of the data source s, qs the proportion of reported cases that have a valid diagnostic code in data source s, and  rs,v the proportion of non-missing values for the variable v in data source s [9].

### Statistical analysis

The weekly time series of ILI proportions (SurSaUD) and incidence rates (Sentinelles), for all ages, were analysed with statistical outbreak detection methods, separately for each geographical area. To allow comparisons, each method is tuned with the same set of parameters for all data sources and areas. We briefly describe the methods used below. All statistical analyses were performed with R [10].

Periodic regression

This method has been used by many authors to predict the baseline level of an epidemiological time series in the absence of influenza, and to flag values significantly above this baseline as a possible epidemic [3,11,12]. We modelled the baseline level with a multivariable linear regression model using the following equation:

### Formula 1

E(yw)=α0 + α1w + γ1cos(2πw/52.17) + δ1sin(2πw/52.17) + γ2cos(4πw/52.17) + δ2sin(4πw/52.17),

where yw is, for week w, the proportion of visits for ILI among all visits with a valid diagnosis code (for ED and SOS Médecins data), or the incidence rate for 100,000 inhabitants (for Sentinelles data).

The predicted baseline level $y ^ w$ in a given week w was obtained by fitting the model to the observations of the past 5 years (i.e. 261 weeks: from week w − 261 to week w − 1) that were below their 85% percentile. This trimming was an attempt to ensure that the dataset used to model the baseline level was free of influenza. An observation was flagged as possibly indicative of an epidemic if it exceeded the upper limit of the 95% bilateral prediction interval of the predicted baseline.

Robust periodic regression

Robust periodic regression resembles periodic regression but does not require trimming [13]. Instead, an iterative fitting algorithm allows small weights to be attributed to the highest values of the dataset (those which indicate a possible influenza epidemic). We used the same model equation and the same model training period (week w − 261 to week w − 1) as for the periodic regression method. Similarly, an observation was flagged as possibly indicative of an epidemic if it exceeded the upper limit of the 95% prediction interval of the predicted baseline. We employed the rlm function of the R package MASS [14].

Hidden Markov model

We used a two-state hidden Markov model to differentiate between epidemic and non-epidemic weeks [15-18]. In our model, under state 1 (non-epidemic state), the observations were normally distributed with mean µ1 and variance σ12; under state 2 (epidemic state), the observations were normally distributed with mean µ2 and variance σ22.

Before fitting the model, the missing values occurring between two observations were imputed from the mean of these two observations. The model was then applied sequentially to infer the state of each observation based on the given observation plus the preceding 260 observations. This subset of weeks was first used to estimate µ1 (respectively µ2) as the mean of the observations below (upper) the 80% percentile, and σ1222) as the variance of the observations below (upper) the 80% percentile. The state of the observation of interest was inferred by fitting the model with maximum likelihood techniques, using the R package HiddenMarkov [19].

Synthesis of statistical alarms

An influenza alarm level was created for every week in each area by combining the results of the detection methods as follows:

### Formula 2

where p = na/n is based on n, the number of statistical results (= 9 when the three detection methods provided a result for all three data sources, and < 9 if one or more data sources are missing or if one or more of the methods fail) and na the number of epidemic alarms (between 0 and n).

The cut-off values were determined from historical data, under the following two constraints: They should (i) avoid issuing non-epidemic or post-epidemic alarm levels in the ascending phase of the epidemic and (ii) ensure that pre-epidemic alarms were issued.

### Web application

The development of MASS started in June 2015, based on the shiny R package [20]. At the end of July, a video demonstrating the use of the prototype was sent to 31 epidemiologists, in the regional units and at the national level, along with a web questionnaire asking for their opinions and suggestions for improvements. The answers influenced the development of MASS. The application was launched in December 2015, and refined at different moments during the following months. MASS can be accessed from within the national public health agency headquarters or from regional units, in the latter case via a secured network. The schematic process is illustrated in Figure 1.

The interface is organised in tab panels. Dropdown menus and radio buttons on the left hand side can be used to filter the data to display by syndromic group, data source, age group, area, date and time step. In particular, the ‘Data’ panel offers dynamic charts, tables, gauges and maps, and the ‘Detection of epidemic periods’ panel presents statistical alarms and alarm levels (Figure 2). The R code in MASS is available on GitHub (https://github.com/cpelat/MASS) and a demonstration is available at https://cpelat.shinyapps.io/mass/.

Every week during the 2015/16 influenza surveillance period, the epidemiologists in the national public health agency’s regional units were asked to choose an alert level for the region(s) they were in charge of: non-epidemic, pre-/post-epidemic or epidemic. The epidemiologists based their analysis on the MASS alarm levels and on other relevant information such as virological data or local surveillance systems, including the surveillance of clusters of acute respiratory infections in nursing homes. Their decision about which level to report could also take into account the precision of each data source in their region and the alarm level in neighbouring regions.

Every Tuesday morning, the regional epidemiologists entered their chosen alert level(s) in a dedicated web questionnaire and, if appropriate, the reason why they proposed an alert level different from the MASS alarm level. The proposed alert levels were reviewed collectively at the national public health agency’s headquarters every Tuesday at noon and discussed in the weekly conference call that was held every Tuesday at 16:00 with the national surveillance partners (e.g. virologists from the influenza reference centres, epidemiologists and virologists from the Sentinelles network). The validated alert levels were published on Wednesdays in the weekly national influenza report posted on the agency website. The regional units of the national public health agency informed their regional health agency and communicated alert levels through their regional epidemiological reports.

## Results

### The 2015/16 influenza epidemic

The new procedure for influenza alert in metropolitan France was implemented from week 2015-W50 to 2016-W17. Among the 2,144 viruses isolated in that season in ambulatory medicine through random sampling of ILI cases, 70% were influenza B viruses (lineage B/Victoria), and 27% belonged to the subtype A(H1N1)pdm09 [21]. During this period, participating ED staff reported 52,271 ILI cases, SOS Médecins’ associations 125,087 cases and Sentinelles GPs 11,298 cases. Figure 3 presents the distribution of ILI cases by age group, sex and whether hospitalisation followed the visit or was encouraged.

MASS was used every week throughout this period by the epidemiologists involved in influenza surveillance at the national public health agency’s regional and national levels. The influenza alert levels published in the weekly reports are shown in Figure 4, panel A, along with the MASS alarm levels, to highlight the discrepancies between them. The first region to signal an epidemic was Brittany, in week 2016-W03, with 11 other regions at a pre-epidemic alert level. All regions were at the epidemic alert level within the subsequent 2 weeks (except Corsica, 3 weeks). MASS emitted the first epidemic alarm level for the area of metropolitan France in week 2016-W04, and the epidemic was declared at the national level in the same week.

All French metropolitan regions except Corsica remained at epidemic alert level until week 2016-W14, although MASS emitted post-epidemic alarms levels for six regions in that same week. Most regions then transited through the post-epidemic alert level to the non-epidemic level, and the epidemic was declared over in all regions of metropolitan France in week 2016-W17. At the level of metropolitan France, the post-epidemic alarm level was signalled and epidemiologically validated in week 2016-W15. The non-epidemic alarm level was signalled in week 2016-W16 and the end of the national epidemic was declared.

### Performance

The alarm level was based on nine statistical results for 381 of the 462 analysed region-weeks (22 regions x 21 weeks), and on eight statistical results for the other 81 region-weeks because of failures adjusting the robust periodic regression model to the Sentinelles time series in four regions: Picardy (21 failures over the 21 weeks), Burgundy (21 failures), Upper Normandy (21 failures) and Lower Normandy (18 failures). In these regions, Sentinelles ILI incidence rate was zero for more than half of the weeks (vs less than half for the other regions).

Discrepancies between the MASS alarm levels and the epidemiologists’ decisions about levels accounted for less than 10% of the decisions (44/462): nine pre-/post-epidemic alarm levels from MASS were transformed into non-epidemic alert levels, 29 into epidemic alert levels; four epidemic and two non-epidemic alarm levels from MASS were transformed into pre-/post-epidemic alert levels.

### Sensitivity analysis

We recalculated the regional alarm levels for the influenza surveillance period 2015/16 using all possible subsets of data sources and outbreak detection methods, and counted the discrepancies with the epidemiologists’ decisions (Table). The best alternative, generating only 36 discrepancies, was based on data from ED and SOS Médecins only with the periodic regression and robust periodic regression models (Figure 4, panel B).

### Table

Number of discrepancies between epidemiologists’ decisions concerning influenza alert level and the alarm level calculated using different subsets of data sources and outbreak detection methods, among the 462 region-weeks of the surveillance period (22 regions x 21 weeks).

Data source(s) 1 method 2 methods 3 methods
PR Robust PR HMM PR and robust PR PR and HMM Robust PR and HMM PR and robust PR and HMM
ED NC NC NC NC NC NC 65
SOS Médecins NC NC NC NC NC NC 59
Sentinelles NC NC NC NC NC NC 86a
ED and SOS Médecins NC NC NC 36 44 76 39
ED and Sentinelles NC NC NC 62 a 69 72 a 67 a
SOS Médecins and Sentinelles NC NC NC 64 a 69 76 a 74 a
ED and SOS Médecins and Sentinelles 56 55a 63 45 a 47 57 a 44 a

ED: emergency department; HMM: Hidden Markov Model; PR: periodic regression model.

NC: the alarm level was not computed for those combinations, which provide at most one or two results to combine.

a The robust PR model applied on the Sentinelles time series failed to provide a result for 81 region-weeks. In those 81 occasions, the alarm level was based on n − 1 statistical results (n being the number of statistical methods multiplied with the number of data sources).

Periodic regression and robust periodic regression models outperformed Hidden Markov Models, both when used alone and in combinations. We explored the cause for discrepancy for each method when used alone for all three data sources. For the periodic regression model, 51 of 56 discrepancies were due to the epidemiologists choosing an alert level higher than the alarm level. In contrast, most of the discrepancies between the epidemiologists’ alert level and the alarm levels produced by the robust periodic regression model and the Hidden Markov were the result of epidemiologists choosing a lower alert level (41/55 and 51/63 discrepancies, respectively).

## Discussion

For the first time, our agency was able to announce the start of the influenza epidemic at the regional level, based on a standardised procedure across regions, coordinated at the national level. The lag between the first and last regions to enter the epidemic phase was 3 weeks. This asynchronicity and the fact that public health decisions concerning influenza are taken by regional authorities are a reminder that monitoring influenza at the regional level is essential.

The regional health agencies were informed of the advent of the pre-epidemic phase, then of the epidemic phase, which enabled hospitals to progressively adapt the healthcare provision if needed. Overall, MASS proved useful for local health service planning, even though the impact on healthcare services of the influenza epidemic in 2015/16 was less marked than in 2014/15.

### User feedback

User feedback on the new influenza alert procedure was positive. Epidemiologists appreciated the increased inter-regional and national–regional coordination, where MASS proved pivotal. The fact that MASS was developed in close collaboration with its end users probably contributed to its rapid integration into routine surveillance activities. From our experience, the shiny R package is well suited for building specific reporting tools that need frequent adaptation to users’ needs.

### Agreement between MASS and epidemiologists

The level of agreement between MASS alarm levels and epidemiological decisions regarding alert levels was high during the surveillance period. The changes made by the epidemiologists aimed at correcting levels that seemed unlikely or at following the expected alert level sequence.

With respect to the first point, one example is where MASS signalled an early pre-epidemic alarm level for Poitou-Charentes in week 2015-W51, when all the other regions were non-epidemic. The epidemiologists maintained the non-epidemic alert level. History proved them right as MASS classified the subsequent weeks as non-epidemic. Similarly, MASS signalled a post-epidemic alarm level in four regions after only 1–3 epidemic weeks but the epidemiologists maintained the epidemic alert level because such a short epidemic was unlikely. Here again, history proved them right as MASS classified the subsequent weeks as epidemic.

With respect to the latter point, the expected alert level sequence was: ‘non-epidemic/pre-epidemic/epidemic/post-epidemic/non-epidemic’. For example, in the Ile-de-France and Languedoc-Roussillon regions, a non-epidemic MASS alarm level directly followed the epidemic level and was transformed by epidemiologists into a post-epidemic level. However, in some instances (e.g. week 2016-W02 in Brittany), the epidemiologists preferred not to validate the pre-epidemic alarm level in the MASS sequence, maybe because of doubts about the application’s functioning or about the precision of some data sources at that time. As a result, Brittany transited directly from the non-epidemic alert level (week 2016-W02) to the epidemic alert level (week 2016-W03).

As pre- (and post-) epidemic phases are important for health service preparedness, we evaluated how well MASS and the epidemiologists detected them: MASS bypassed the pre-epidemic phase in one region (vs three for the epidemiologists) and the post-epidemic phase in three regions (vs two for the epidemiologists).

### Limitations and perspectives

Although MASS produced valuable results for routine work, areas of improvement were identified. Firstly, for the sake of simplicity, all data sources and detection methods shared the same weight in the construction of MASS alarm levels: e.g. three alarms on the same data source are equivalent to one alarm for each source. Alternative criteria could be considered here, for example the number of sources with at least one alarm. Besides, in a sensitivity analysis, the alarm level that was closest to the epidemiologists’ decisions relied only on two data sources and two methods. This alternative construction could be considered for future seasons, although it was decided to keep the current construction for the 2016/17 season. Furthermore, the different data sources are associated with different levels of precision, since the number of ILI cases reported by SOS Médecins’ GPs is twice that reported by ED and 10 times that reported by Sentinelles GPs. They also serve different case populations in terms of ILI definition, age distribution (with a higher proportion of children younger than 5 years in ED data and of young active people in SOS Médecins’ data) and severity of symptoms (hospitalisation after visit being much more common in ED), while the sex ratio is ca 1:1 in all surveillance systems. All these criteria are considered by the epidemiologists when making their decision but are not yet included in the automatic construction of MASS alarm levels. This could constitute a possible evolution of the tool, although recent analyses indicate that the dynamics of most influenza epidemics vary little between age groups [22]. In theory, a patient could be reported in the three detection systems, and in multiple weeks, but this is unlikely. As the data were anonymised before transmission, we had no means to correct for that possibility.

Future developments include the integration of virological data in MASS.

Finally, several countries have been using an influenza epidemic threshold for years [23,24], yet the best data source and the best outbreak detection method to use are still a matter of discussion [25]. We shall continue to look for the combination of data sources and methods that produces the most relevant results for our needs.

### The Regional Influenza study group

At Santé publique France, French national public health agency, Saint-Maurice, France, Regional units (Cire): Olivier Retel (Cire Bourgogne-Franche-Comté), Bruno Hubert (Cire Pays de la Loire), Franck Golliot (Cire Occitanie), Lisa King (Cire Bretagne), Ibrahim Mounchetrou Njoya (Cire Ile-de-France), Christine Saura (Cire Auvergne-Rhône-Alpes), Laurent Filleul (Cire Océan Indien).

### Acknowledgements

We thank the epidemiologists of Santé publique France implicated in influenza surveillance and the partners involved in the surveillance systems described in this paper: hospital emergency department staff from the OSCOUR® network, SOS Médecins association members and the GPs and epidemiologists of the Sentinelles network. We are also thankful to our colleague Laurent Marie for installing a shiny server for MASS and ensuring that it functioned adequately.

None declared.

### Authors’ contributions

IB and DLB were in charge of piloting influenza surveillance for Santé publique France, the French national public health agency. The epidemiologists of the Regional Influenza study group participated in reflexions about how to improve the influenza alert procedure and about the features needed in the web application MASS. MR, AF and CCS were responsible for coordinating the syndromic surveillance system SurSaUD. CP and YLS analysed the surveillance data and programmed MASS.

CP wrote the initial draft of the manuscript. CP, MR, AF, CCS, IB, DLB, YLS and the Regional Influenza study group commented on this, and subsequent drafts, and approved the final draft for submission.

## References

1. Influenza surveillance teams. Surveillance de la grippe en France métropolitaine. Saison 2014-2015.[Influenza activity in mainland France: 2014-15 season]. Bull Epidemiol Hebd (Paris). 2015;32-33:593-8.

2. Valleron AJ, Garnerin P. Computerised surveillance of communicable diseases in France.Commun Dis Rep CDR Rev. 1993;3(6):R82-7.PMID: 7693158

3. Costagliola D, Flahault A, Galinec D, Garnerin P, Menares J, Valleron AJ. A routine tool for detection and assessment of epidemics of influenza-like syndromes in France.Am J Public Health. 1991;81(1):97-9. DOI: 10.2105/AJPH.81.1.97 PMID: 1983924

4. Caserio-Schönemann C, Bousquet V, Fouillet A, Henry V, for the SurSaUD team. Le système de surveillance syndromique SurSaUD. [The French syndromic surveillance system SUrSaUD]. Bull Epidemiol Hebd (Paris). 2014;3-4:38-44. French.

5. Caserio-Schönemann C, Meynard JB. Ten years experience of syndromic surveillance for civil and military public health, France, 2004-2014.Euro Surveill. 2015;20(19):21126. DOI: 10.2807/1560-7917.ES2015.20.19.21126 PMID: 25990360

6. Josseran L, Nicolau J, Caillère N, Astagneau P, Brücker G. Syndromic surveillance based on emergency department activity and crude mortality: two examples.Euro Surveill. 2006;11(12):225-9.PMID: 17370967

7. Réseau Sentinelles. Bilan annuel 2015. [Annual report 2015.] Paris: Institut Pierre Louis d'Epidémiologie et de Santé Publique; 2016. French. Available from: http://www.sentiweb.fr/document/3583

8. Souty C, Turbelin C, Blanchon T, Hanslik T, Le Strat Y, Boëlle PY. Improving disease incidence estimates in primary care surveillance systems.Popul Health Metr. 2014;12(1):19. DOI: 10.1186/s12963-014-0019-8 PMID: 25435814

9. Schaeffer R, Mendenhall W, Ott L. Elementary survey sampling. 3rd ed. Boston: Prindle, Weber and Schmidt; 1986.

10. R Core Team. R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing; 2015. Available from: http://www.R-project.org/

11. Serfling RE. Methods for current statistical analysis of excess pneumonia-influenza deaths.Public Health Rep. 1963;78(6):494-506. DOI: 10.2307/4591848 PMID: 19316455

12. Pelat C, Boëlle PY, Cowling BJ, Carrat F, Flahault A, Ansart S,  et al.  Online detection and quantification of epidemics. BMC Med Inform Decis Mak. 2007;7(1):29. DOI: 10.1186/1472-6947-7-29 PMID: 17937786

13. Muscatello DJ, Cretikos MA, Macintyre CR. All-cause mortality during first wave of pandemic (H1N1) 2009, New South Wales, Australia, 2009.Emerg Infect Dis. 2010;16(9):1396-402. DOI: 10.3201/eid1609.091723 PMID: 20735923

14. Venables WN, Ripley BD. Modern applied statistics with S. 4th ed. New York: Springer; 2002.

15. Le Strat Y, Carrat F. Monitoring epidemiologic surveillance data using hidden Markov models.Stat Med. 1999;18(24):3463-78. DOI: 10.1002/(SICI)1097-0258(19991230)18:24<3463::AID-SIM409>3.0.CO;2-I PMID: 10611619

16. Martínez-Beneito MA, Conesa D, López-Quílez A, López-Maside A. Bayesian Markov switching models for the early detection of influenza epidemics.Stat Med. 2008;27(22):4455-68. DOI: 10.1002/sim.3320 PMID: 18618414

17. Rath TM, Carreras M, Sebastiani P. Automated detection of influenza epidemics with Hidden Markov Models. Lect notes comput sc. 2003;2810:521-32.

18. Watkins RE, Eagleson S, Veenendaal B, Wright G, Plant AJ. Disease surveillance using a hidden Markov model.BMC Med Inform Decis Mak. 2009;9(1):39. DOI: 10.1186/1472-6947-9-39 PMID: 19664256

19. Harte D. HiddenMarkov: Hidden Markov models. R package version 1.8-7. Wellington2016. Available from: https://cran.r-project.org/web/packages/HiddenMarkov/HiddenMarkov.pdf

20. Chang W, Cheng J, Allaire J, Xie Y, McPherson J. shiny: web application framework for R. R package version 0.13.0. 2016. Available from: https://CRAN.R-project.org/package=shiny

21. Influenza surveillance teams. Surveillance de la grippe en France métropolitaine, saison 2015-2016. [Influenza activity in mainland France, season 2015-2016]. Bull Epidemiol Hebd (Paris). 2016;32-33:558-63. French.

22. Fleming DM, Durnall H, Warburton F, Ellis JS, Zambon MC. Is the onset of influenza in the community age-related?Epidemiol Infect. 2016;144(11):2295-305. DOI: 10.1017/S0950268816000510 PMID: 27350234

23. Vega T, Lozano JE, Meerhoff T, Snacken R, Mott J, Ortiz de Lejarazu R,  et al.  Influenza surveillance in Europe: establishing epidemic thresholds by the moving epidemic method. Influenza Other Respi Viruses. 2013;7(4):546-58. DOI: 10.1111/j.1750-2659.2012.00422.x PMID: 22897919

24. Green HK, Charlett A, Moran-Gilad J, Fleming D, Durnall H, Thomas DR,  et al.  Harmonizing influenza primary-care surveillance in the United Kingdom: piloting two methods to assess the timing and intensity of the seasonal epidemic across several general practice-based surveillance schemes. Epidemiol Infect. 2015;143(1):1-12. DOI: 10.1017/S0950268814001757 PMID: 25023603

25. Schanzer DL, Saboui M, Lee L, Domingo FR, Mersereau T. Leading Indicators and the Evaluation of the Performance of Alerts for Influenza Epidemics.PLoS One. 2015;10(10):e0141776. DOI: 10.1371/journal.pone.0141776 PMID: 26513364

 Tweet

Disclaimer: The opinions expressed by authors contributing to Eurosurveillance do not necessarily reflect the opinions of the European Centre for Disease Prevention and Control (ECDC) or the editorial team or the institutions with which the authors are affiliated. Neither ECDC nor any person acting on behalf of ECDC is responsible for the use that might be made of the information in this journal. The information provided on the Eurosurveillance site is designed to support, not replace, the relationship that exists between a patient/site visitor and his/her physician. Our website does not host any form of commercial advertisement. Except where otherwise stated, all manuscripts published after 1 January 2016 will be published under the Creative Commons Attribution (CC BY) licence. You are free to share and adapt the material, but you must give appropriate credit, provide a link to the licence, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.