From Concept to Reality: The Journey of Creating the FPR Modeller

In the world of healthcare and medical professionals, understanding the intricacies of pay progression and restoration is crucial. This need inspired me to develop the Full Pay Restoration (FPR) Modeller, a sophisticated web application designed to help medical professionals, union representatives, and policymakers visualize and analyze complex pay scenarios. In this post, I’ll take you through the development process, highlight key features, and share insights into the technology that powers this tool.

The Spark: Why Build an FPR Modeller?

The idea for the FPR Modeller stemmed from the ongoing discussions about pay restoration in the medical field. I realized there was a need for a tool that could:

1. Visualize pay progression over time
2. Calculate the impact of inflation on real pay
3. Track progress towards full pay restoration
4. Estimate the costs associated with different pay deals

With these goals in mind, I set out to create a user-friendly, data-driven application that could handle these complex calculations and present the results in an easily digestible format.

Choosing the Right Tools

After considering various options, I decided to build the FPR Modeller using:

• Python: For its robust data processing capabilities and extensive libraries
• Streamlit: A powerful framework for creating web applications with Python
• Plotly: For interactive and visually appealing charts
• Pandas: For efficient data manipulation and analysis

These tools allowed me to rapidly develop a prototype and iterate on the design based on user feedback.

Key Features of the FPR Modeller

Let’s dive into some of the core features I implemented in the app:

1. Flexible Input Parameters

Users can adjust various parameters, including:

• Inflation measure (RPI or CPI)
• FPR calculation period
• Number of years in the pay deal
• Doctor counts at each nodal point
• Year-by-year pay rises and inflation projections

This flexibility allows for a wide range of scenarios to be modeled and compared.

Here’s a glimpse into how the sidebar inputs are set up using Streamlit:

def setup_sidebar():
    initialize_session_state()
    
    st.sidebar.title("Modeller Settings ⚙️")
    
    st.sidebar.subheader("Calculate Pay Restoration Targets.")
    
    inflation_type = st.sidebar.radio("Select inflation measure:", ("RPI", "CPI"), key="inflation_type", on_change=update_fpr_targets, horizontal=True)
    
    col1, col2 = st.sidebar.columns(2)
    with col1:
        fpr_start_year = st.selectbox(
            "FPR start year",
            options=AVAILABLE_YEARS,
            index=AVAILABLE_YEARS.index(st.session_state.fpr_start_year),
            key="fpr_start_year",
            on_change=update_end_year_options
        )
    with col2:
        fpr_end_year = st.selectbox(
            "FPR end year",
            options=st.session_state.end_year_options,
            index=st.session_state.end_year_options.index(st.session_state.fpr_end_year),
            key="fpr_end_year",
            on_change=update_fpr_targets
        )
    
    # ... [rest of the function]

    return inflation_type, fpr_start_year, fpr_end_year, num_years, st.session_state.fpr_targets, st.session_state.doctor_counts, year_inputs, additional_hours, out_of_hours

This function sets up the sidebar inputs and manages the state of the application, allowing for dynamic updates based on user interactions.

2. Interactive Visualizations

One of the most powerful aspects of the FPR Modeller is its ability to visually represent complex data. I implemented several interactive charts using Plotly:

• Pay Progression Chart: Shows baseline pay, pay increases, FPR progress, and pay erosion over time
• Pay Increase Curve: Displays nominal increases, real increases, and cumulative costs

These visualizations help users quickly grasp the long-term implications of different pay scenarios.

The creation of interactive visualizations is powered by Plotly. Here’s an example of how the pay progression chart is created:

def create_pay_progression_chart(result, num_years):
    years = [f"Year {i} ({2023+i}/{2024+i})" for i in range(num_years + 1)]
    nominal_pay = result["Pay Progression Nominal"]
    baseline_pay = result["Base Pay"]
    pay_increase = [max(0, pay - baseline_pay) for pay in nominal_pay]
    percent_increase = [(increase / baseline_pay) * 100 for increase in pay_increase]
    pay_erosion = [-(x) * 100 for x in result["Real Terms Pay Cuts"]]
    fpr_progress = result["FPR Progress"]

    fig = make_subplots(specs=[[{"secondary_y": True}]])

    fig.add_trace(
        go.Bar(x=years, y=[baseline_pay] * len(years), name="Baseline Pay", marker_color='rgb(0, 123, 255)'),
        secondary_y=False,
    )

    fig.add_trace(
        go.Bar(x=years, y=pay_increase, name="Pay Increase", marker_color='rgb(255, 165, 0)',
               hovertemplate='Year: %{x}<br>Total Pay: £%{customdata[0]:,.2f}<br>Increase: £%{y:,.2f} (%{customdata[1]:.2f}%)<extra></extra>',
               customdata=list(zip(nominal_pay, percent_increase))),
        secondary_y=False,
    )

    fig.add_trace(
        go.Scatter(x=years, y=fpr_progress, name="FPR Progress", line=dict(color='rgb(0, 200, 0)', width=2)),
        secondary_y=True,
    )

    fig.add_trace(
        go.Scatter(x=years, y=pay_erosion, name="Pay Erosion", line=dict(color='rgb(255, 0, 0)', width=2)),
        secondary_y=True,
    )

    # ... [layout configuration]

    return fig

This function demonstrates how multiple data series are combined into a single, informative chart using Plotly’s make_subplots and various trace types.

3. Detailed Cost Breakdown

Understanding the financial impact of pay deals is crucial. The app provides a comprehensive cost breakdown, including:

• Basic pay costs
• Pension contributions
• Additional hours and out-of-hours costs
• Employer National Insurance contributions
• Tax recouped

This level of detail allows for more informed decision-making in pay negotiations.

The cost breakdown calculations involve several components. Here’s a snippet showing how the costs are calculated:

def calculate_costs(pay_progression_nominal, doctor_count, year_inputs, name, post_ddrb_pay, additional_hours, out_of_hours):
    yearly_basic_costs = []
    yearly_total_costs = []
    yearly_tax_recouped = []
    yearly_net_costs = []
    yearly_employer_ni_costs = []
    yearly_pension_costs = []

    def calculate_total_pay(basic_pay):
        additional_pay = (basic_pay / 40) * additional_hours
        ooh_pay = (basic_pay / 40) * out_of_hours * 0.37
        return basic_pay, additional_pay, ooh_pay

    def calculate_tax(total_pay, basic_pay):
        pension_contribution = calculate_pension_contribution(basic_pay)
        taxable_pay = total_pay - pension_contribution
        income_tax = calculate_income_tax(taxable_pay)
        ni = calculate_national_insurance(taxable_pay)
        return income_tax, ni, pension_contribution

    for year, (current_pay, prev_pay) in enumerate(zip(pay_progression_nominal[1:], pay_progression_nominal)):
        current_basic, current_additional, current_ooh = calculate_total_pay(current_pay)
        current_total_pay = current_basic + current_additional + current_ooh
        current_income_tax, current_ni, current_pension = calculate_tax(current_total_pay, current_basic)

        # ... [cost calculations for each year]

        yearly_basic_costs.append(basic_pay_cost)
        yearly_total_costs.append(total_cost)
        yearly_tax_recouped.append(tax_recouped)
        yearly_net_costs.append(net_cost)
        yearly_employer_ni_costs.append(employer_ni_cost)
        yearly_pension_costs.append(pension_cost)

    return yearly_basic_costs, yearly_total_costs, yearly_tax_recouped, yearly_net_costs, yearly_employer_ni_costs, yearly_pension_costs

This function showcases how various components of pay and taxation are factored into the cost calculations, providing a comprehensive breakdown of the financial implications of pay deals.

4. FPR Achievement Tracking

The app calculates and displays whether the proposed pay deal achieves full pay restoration for each nodal point. This feature helps users quickly assess the effectiveness of different scenarios in meeting FPR targets.

The FPR achievement is calculated and displayed using the following function:

def display_fpr_achievement(results):
    st.subheader(":blue-background[👈 Will FPR be achieved from this pay deal? 🕵️]")
    fpr_achieved = all(result["FPR Progress"][-1] >= 100 for result in results)
    
    if fpr_achieved:
        st.success("Yes, FPR will be achieved for all nodal points.")
    else:
        st.error("No, FPR will not be achieved for all nodal points. But some progress has been made...  \nNote the residual pay erosion figures in % below.")
    
    cols = st.columns(len(results))
    for i, result in enumerate(results):
        with cols[i]:
            fpr_progress = result["FPR Progress"][-1]
            pay_erosion = result["Real Terms Pay Cuts"][-1]
            
            pay_erosion_formatted = f"{pay_erosion * 100:.2f}%"
            fpr_progress_formatted = f"FPR: {fpr_progress:.2f}%"
            
            st.metric(
                label=f"{result['Nodal Point']}",
                value=pay_erosion_formatted,
                delta=fpr_progress_formatted,
                delta_color="normal"
            )

This function not only determines whether FPR is achieved but also presents the results in an easy-to-understand format using Streamlit’s metric component.

The Heart of the Calculations: FPR and Pay Erosion

At the core of the FPR Modeller are the calculations for Full Pay Restoration progress and pay erosion. Let’s take a closer look at how these are implemented:

def calculate_real_terms_change(pay_rise, inflation):
    return ((1 + pay_rise) / (1 + inflation)) - 1

def calculate_new_pay_erosion(current_erosion, real_terms_change):
    return ((1 + current_erosion) * (1 + real_terms_change)) - 1

def calculate_fpr_and_erosion(base_pay, pay_progression_nominal, pay_progression_real, fpr_percentage, year_inputs):
    real_terms_pay_cuts = [-fpr_percentage / 100]
    fpr_progress = [0]

    for year, (nominal_pay, real_pay, year_input) in enumerate(zip(pay_progression_nominal[1:], pay_progression_real[1:], year_inputs)):
        total_pay_rise = (nominal_pay - pay_progression_nominal[year]) / pay_progression_nominal[year]
        inflation_rate = year_input["inflation"]
        
        real_terms_change = calculate_real_terms_change(total_pay_rise, inflation_rate)
        current_pay_cut = calculate_new_pay_erosion(real_terms_pay_cuts[-1], real_terms_change)
        
        real_terms_pay_cuts.append(current_pay_cut)
        current_progress = (fpr_percentage / 100 + current_pay_cut) / (fpr_percentage / 100) * 100
        fpr_progress.append(current_progress)

    return real_terms_pay_cuts[1:], fpr_progress[1:]

These functions demonstrate how the app calculates real terms changes in pay, tracks pay erosion over time, and measures progress towards the FPR target.

Overcoming Challenges

Developing the FPR Modeller wasn’t without its challenges. Some of the key hurdles I faced and overcame included:

1. Complex Calculations: Ensuring accuracy in the various interrelated calculations required careful planning and extensive testing.

2. Performance Optimization: With multiple interactive elements and real-time calculations, optimizing performance was crucial for a smooth user experience.

3. User Interface Design: Balancing the need for detailed inputs with a clean, intuitive interface required several iterations.

4. Data Visualization: Choosing the right types of charts and ensuring they effectively communicated the key insights took experimentation and user feedback.

The Development Process

Here’s a brief overview of my development process:

1. Requirements Gathering: I started by clearly defining the app’s requirements based on the needs of potential users.

2. Prototype Development: Using Streamlit, I quickly built a basic prototype to test the core functionality.

3. Iterative Refinement: I continuously improved the app based on self-testing and feedback, adding features and refining the user interface.

4. Testing and Validation: Rigorous testing was performed to ensure the accuracy of calculations and the reliability of the app across different scenarios.

5. Documentation and Deployment: Finally, I created user documentation and deployed the app for wider access.

Future Enhancements

While the current version of the FPR Modeller is fully functional, there’s always room for improvement. Some ideas for future enhancements include:

1. Integration with live economic data sources for up-to-date inflation figures
2. Additional visualization options, such as comparative charts for different scenarios
3. Export functionality for reports and data
4. Collaborative features allowing multiple users to work on scenarios together

Conclusion

Developing the FPR Modeller has been an exciting journey, blending data science, web development, and domain-specific knowledge in healthcare economics. This project showcases the power of modern web technologies and data visualization techniques in creating practical tools for complex real-world problems.

I hope this behind-the-scenes look at the FPR Modeller inspires you to tackle similar challenges in your field. Whether you’re a developer, a data scientist, or simply someone interested in the intersection of technology and specialized domains, there’s always an opportunity to create tools that can make a difference.

Feel free to check out the FPR Modeller and let me know your thoughts or suggestions for improvement. Happy modeling!

Have you ever wondered if those sunny days actually boost your mood, or if rainy days truly bring you down? This age-old question has puzzled scientists, philosophers, and just about everyone else for generations. But what if we could use data to finally find an answer? In this post, I’ll dive into the fascinating intersection of meteorology and psychology to explore whether the weather really has a tangible effect on our happiness. By leveraging data science, I aim to uncover patterns and insights that could shed light on how our environment influences our well-being. So, let’s embark on this journey to see if we can decode the weather’s impact on happiness using the power of data.

For the full analysis and more details, visit this link.

Part 0: Data Cleaning and Preparation

To begin, I needed comprehensive weather data from various stations across the UK. I wrote a Python script to download and process the data from the MetOffice, ensuring it was ready for analysis. Here’s a glimpse of the data preparation process:

import os
import requests

folder_name = '/path/to/weather_data'
os.makedirs(folder_name, exist_ok=True)

def download_file(station_name):
    url = f'http://www.metoffice.gov.uk/pub/data/weather/uk/climate/stationdata/{station_name}data.txt'
    response = requests.get(url)
    response.raise_for_status()
    save_path = os.path.join(folder_name, f'{station_name}data.txt')
    with open(save_path, 'wb') as file:
        file.write(response.content)

with open('data/stations.txt', 'r') as file:
    station_names = file.read().splitlines()

for station_name in station_names:
    download_file(station_name)

This script ensured I had all the necessary weather data, which was then processed into a structured format using pandas.

I then used R for more sophisticated data wrangling, converting coordinates and validating the data:

library(tidyverse)
library(sf)

S_weather <- read_csv('path/to/combined_weather_data.csv')

# Convert to spatial dataframe
weather_sf <- st_as_sf(S_weather, coords = c("easting", "northing"), crs = 27700) %>%
  st_transform(crs = 4326)

# Extract coordinates
coords <- st_coordinates(weather_sf)
S_weather <- S_weather %>% mutate(Longitude = coords[, 1], Latitude = coords[, 2])

# Data validation
validate_data <- function(df) {
  df %>% 
    filter(year >= 1850 & year <= 2024,
           month >= 1 & month <= 12,
           latitude >= 49.9 & latitude <= 60.9,
           longitude >= -8 & longitude <= 2,
           altitude >= -2.75 & altitude <= 1343,
           t_min >= -50 & t_min <= 50,
           t_max >= -50 & t_max <= 50,
           rain >= 0 & rain <= 1000,
           sun >= 0 & sun <= 1000)
}

S_weather <- validate_data(S_weather)

Part 1: Clustering by Weather Data

Once I had the data, I performed clustering to identify patterns. Using k-means clustering, I grouped weather stations based on key weather parameters like minimum temperature, maximum temperature, rainfall, sunshine duration, and air frost days.

library(tidyverse)
library(cluster)

set.seed(123)
scaled_data <- scale(S_weather %>% select(t_min, t_max, rain, sun, af_inmth))

wss <- sapply(2:15, function(k){
  sum(kmeans(scaled_data, centers=k, nstart=25)$withinss)
})

plot(2:15, wss, type='b', xlab='Number of Clusters', ylab='Within groups sum of squares')

I determined the optimal number of clusters using the elbow method and visualised the clusters with ggplot2:

library(ggpubr)

kmeans_result <- kmeans(scaled_data, centers=3, nstart=25)
S_weather <- S_weather %>% mutate(cluster = as.factor(kmeans_result$cluster))

ggscatter(S_weather, x = "sun", y = "af_inmth", color = "cluster", 
          ellipse = TRUE, legend = "right", palette = "jco") +
  ggtitle("Clusters based on Sunshine Duration and Air Frost Days")

Part 2: Predicting Geographic Regions

Next, I used the weather data to predict whether a station is located in the Northern, Middle, or Southern UK. This classification was done using a Random Forest model, implemented in the tidymodels framework.

library(tidymodels)

region_rec <- recipe(region ~ ., data = S_weather) %>% 
  step_impute_knn(all_predictors()) %>% 
  step_zv(all_predictors()) %>% 
  step_normalize(all_predictors())

rf_spec <- rand_forest(trees = 100) %>% 
  set_mode("classification") %>% 
  set_engine("ranger")

region_wf <- workflow() %>% 
  add_recipe(region_rec) %>% 
  add_model(rf_spec)

set.seed(234)
region_folds <- vfold_cv(S_weather, v = 10)
rf_rs <- region_wf %>% fit_resamples(region_folds)

region_pred <- rf_rs %>% collect_predictions() %>% mutate(correct = region == .pred_class)
accuracy <- accuracy(region_pred, truth = region, estimate = .pred_class)

My model showed an impressive accuracy, highlighting the potential of machine learning in interpreting geographic nuances based on meteorological data.

Conclusion

Through this data-driven exploration, I’ve seen that weather data can indeed reveal interesting patterns and even predict geographic locations based on climatic conditions. While this analysis doesn’t directly answer whether weather influences individual happiness, it lays the groundwork for understanding the broader environmental factors at play. Future research could integrate psychological data to directly link weather patterns with mood changes, providing deeper insights into this intriguing question.

Stay tuned as I continue to explore the fascinating connections between our environment and well-being! For the full analysis and more details, visit this link.