AI E-Mail Assistant

Jul 14

Midterm_Mike_Anderson

Alex Carter, a seasoned Senior Manager in Software Development at Orion Tech Solutions, plays a pivotal role in ensuring the smooth execution of multiple IT projects. Orion Tech Solutions, a mid-sized IT services company, prides itself on delivering cutting-edge software solutions to global clients, balancing innovation with operational efficiency.

With years of experience in software engineering and project management, Alex’s day is a mix of strategic planning, problem-solving, and stakeholder coordination. Managing a diverse portfolio of projects, Alex works closely with internal teams, vendors, and clients, ensuring deliverables meet expectations while navigating technical and organizational challenges.

The Challenge

A high-profile client has recently entrusted Orion Tech Solutions with the development of a next-generation cloud security platform. While the project holds immense potential for growth and recognition, it also brings complexity, tight deadlines, and high stakeholder expectations.

As the project nears critical milestones, Alex faces multiple challenges:

Ensuring timely delivery while balancing resource constraints.
Managing escalations related to security vulnerabilities discovered during testing.
Communicating with clients, vendors, and internal teams to align expectations.
Handling unexpected system downtimes due to scheduled IT maintenance.
Addressing compliance concerns raised by the legal team on certain third-party integrations.

With multiple stakeholders relying on Alex’s leadership, clear and effective communication is essential. Every email, meeting, and decision made plays a role in determining the project's success.

The Case Study¶

This case study explores the real-world challenges Alex encounters, highlighting how well-structured communication can help in:

Crisis Management – Handling security breaches and system outages.
Client Relations – Effectively communicating progress and addressing feedback.
Team Coordination – Ensuring alignment between development, security, and operations teams.
Strategic Decision-Making – Navigating legal, financial, and technical roadblocks.

Through realistic scenarios and email exchanges, we will examine how Alex can leverage professional communication strategies to tackle these challenges head-on, ensuring project success and business continuity.

In [1]:

# Imports
import pandas as pd
import json
from openai import OpenAI
import os
from datetime import timedelta
from IPython.display import display, Markdown, HTML
from tqdm import tqdm
import seaborn as sns
from markdown import markdown
import re

In [2]:

# Import e-mails and human labeled e-mails
df = pd.read_csv('Data/Alex_emails_march_04-1.csv', encoding='ANSI')
df_true = pd.read_csv('Data/Alex_emails_march_04-1_labeled.csv', encoding='ANSI')

# Access API Key and use default API Base
with open('Data/config.json', 'r') as f:
    config = json.load(f)
api_key = config['API_KEY']

# Storing API key in environment variable
os.environ['OPENAI_API_KEY'] = api_key

In [3]:

# Initialize OpenAI client
client = OpenAI()
model_name = 'gpt-4o'

In [4]:

# Define the LLM pipeline for prompting
def llm(system_prompt, user_prompt):
  try:
      # Craft the messages to pass to chat.completions.create
      prompt = [
          {'role':'system', 'content': system_prompt},
          {'role': 'user', 'content': user_prompt}
      ]

      response = client.chat.completions.create(
          model=model_name,
          messages=prompt,
          temperature=0
      )

      return response.choices[0].message.content.strip()

  except Exception as e:
      error = f'Sorry, I encountered the following error: \n {e}'
      print(error)

Data Cleaning¶

In [5]:

# Convert dates to datetime and retrieve e-mails from yesterday
df['date_received_dt'] = pd.to_datetime(df['date_received'])
today = pd.to_datetime('2025-03-04')
yesterday = today - timedelta(days=1)
df.loc[df['date_received_dt'] == yesterday, 'email_received'] = 'yesterday'
df.loc[df['date_received_dt'] == today, 'email_received'] = 'today'
df = df.drop(columns=['date_received_dt'])

yesterday_emails = df[df['email_received'] == 'yesterday'].copy()
yesterday_dict = yesterday_emails.to_dict(orient='records')

LLM Output Styling Rules¶

In [46]:

# Styling rules to dictate how the LLM responses should be formatted
style_guide = """MANDATORY STYLING REQUIREMENTS (NEON THEME):
- ALL Normal Paragraph Text MUST use inline HTML with:
  color:#FFFFFF;
  Example: <p style="color:#FFFFFF; ">This is normal text.</p>
- ALL Titles MUST use inline HTML with a cyan glow (#00FFFF), large bold font, and text-shadow for neon glow.
- ALL Bold Numbers MUST use inline HTML with a green glow (#39FF14).
- ALL Important Text MUST use inline HTML with magenta color (#FF1493).
- Horizontal Rules MUST be rendered as green gradient dividers:
  <hr style="border:none;height:2px;background:linear-gradient(to right,#00FF7F,#39FF14);box-shadow:0 0 10px #39FF14;margin:20px 0;">
DO NOT OMIT inline styles. Every single line of text must be wrapped in a styled HTML element.
Plain Markdown is not acceptable. Non-compliance is a critical error.
"""

# AI Assistant Output Function
def ai_display_markdown(markdown_content):
    html_content = markdown(markdown_content)  # Convert Markdown → HTML
    wrapped_html = f"""
    <div style="
        background-color: #21252b;
        padding: 15px;
        border-radius: 10px;
        box-shadow: 0 0 15px #00FFFF;
    ">
        {html_content}
    </div>
    """
    display(HTML(wrapped_html))

Categorization of E-Mails¶

In [7]:

category_system_prompt = """You are an automated secretary for a seasoned Senior Manager in Software Development named Alex Carter and should prioritize emails as such.  I need the attached json of e-mails labeled using the following labels.  Iterate through the JSON object.  I have also provided descriptions of the categories and an example output.

BEGIN CATEGORIES
Urgent and High Priority
Deadline-Driven
Routine Updates and Check-Ins
Non-Urgent and Informational
Personal and Social
Spam and Unimportant
END CATEGORIES

BEGIN DESCRIPTIONS OF CATEGORIES
Urgent and High Priority:
    - Requires Immediate Action
    - Specific prioritization request
    - Urgent in the subject
    - Not a sales pitch
Deadline-Driven:
    - Deadline is Today
    - Must mention a specific deadline
Routine Updates and Check-Ins:
    - Review & Acknowledge
    - Acknowledgement required
    - Approval required
Non-Urgent and Informational:
    - Can Be Deferred or Delegated
    - Simply giving an update or information
    - Main recipient is not Alex Carter
Personal and Social:
    - Work related
    - Social media
    - Optional review
    - Industry sales pitches
Spam and Unimportant:
    - Spam e-mails
    - Non-work related
END DESCRIPTIONS OF CATEGORIES

IMPORTANT: Only assign ONE category per email. If multiple categories apply, choose the one with the HIGHEST priority in this list:
1. Urgent and High Priority
2. Deadline-Driven
3. Routine Updates and Check-Ins
4. Non-Urgent and Informational
5. Personal and Social
6. Spam and Unimportant

Return only the label specified exactly as above, as this will be passed into a dataframe for analysis.

BEGIN CHAIN OF THOUGHT

Is the main recipient Alex Carter?
If no → "Non-Urgent and Informational"

Is there a production degradation or urgent matter that needs attention that need to be solved today?
If yes → "Urgent and High Priority"

Does the email contain:
- A specific date (e.g., March 7) requiring approval or action?
- Phrases like “approve by”, “respond before”, “complete by”, “due on”, “must be done before”, or “to avoid delay”?
- Was a specific date missed or past-due?
If yes → "Deadline-Driven"

Does the e-mail contain information about work progress with no other priority items?
If yes → "Routine Updates and Check-Ins"

Is the main recipient not Alex, regardless of urgency?
If yes → "Non-Urgent and Informational"

Does the e-mail:
- ask about the company's capabilities?
- mention a social network like LinkedIn, Reddit, Facebook, Instagram, Twitter, or X?
If yes → "Personal and Social"

Does the e-mail:
- include information not related to the company or IT?
- ask the recipient to "Act Fast" or include a sales pitch?
If no → "Spam and Unimportant"
END CHAIN OF THOUGHT
"""

In [8]:

yesterday_emails = yesterday_emails.reset_index(drop=True)

# List of possible categories
categories = [
    'Urgent and High Priority',
    'Deadline-Driven',
    'Routine Updates and Check-Ins',
    'Non-Urgent and Informational',
    'Personal and Social',
    'Spam and Unimportant'
]

if 'category' not in yesterday_emails.columns:
    yesterday_emails['category'] = None

# Iterate through e-mails and assign a category
for index, email in tqdm(enumerate(yesterday_dict), total=len(yesterday_dict), desc='Processing Emails'):
    prompt = f"""The following e-mail needs to be analyzed and placed in one of the defined categories:
    *BEGIN EMAIL*
    {json.dumps(email, indent=2)}
    *END EMAIL*
    """

    category_by_llm = llm(category_system_prompt, prompt)

    clean_category = category_by_llm.strip().strip('"').rstrip('.')

    if clean_category in categories:
        yesterday_emails.at[index, 'category'] = clean_category
    else:
        print(f"⚠️ Unexpected category: {repr(category_by_llm)}")
        yesterday_emails.at[index, 'category'] = None

Processing Emails: 100%|██████████| 51/51 [00:56<00:00,  1.10s/it]

In [9]:

# Category Counts
yesterday_emails['category'].value_counts(dropna=False)

Out[9]:

category
Non-Urgent and Informational     16
Urgent and High Priority         14
Routine Updates and Check-Ins     9
Deadline-Driven                   4
Personal and Social               4
Spam and Unimportant              4
Name: count, dtype: int64

Executive E-Mail Summary¶

In [10]:

# Convert e-mails to dictionary format
yesterday_dict = yesterday_emails.to_dict(orient='records')

In [11]:

# Define counts for use in the LLM output
category_counts = yesterday_emails['category'].value_counts()
critical_emails = category_counts['Urgent and High Priority'] + category_counts['Deadline-Driven']
total_emails = len(yesterday_emails)

In [12]:

summary_system_prompt = f"""You are an automated secretary for a seasoned Senior Manager in Software Development named Alex Carter.  Output the following summary using the markdown format specified.  I will also pass in a JSON object for you to parse and collect the relevant information:

{style_guide}

BEGIN CATEGORY COUNTS
## <span style="color:#00FFFF;text-shadow:0 0 10px #00FFFF;">📧 Executive Summary of Emails Received</span>

### Number of Emails from Yesterday: {total_emails}

#### Breakdown of Emails by Category:
🛑 Urgent & High-Priority Emails: {category_counts['Urgent and High Priority']}
⚡ Deadline-Driven Emails: {category_counts['Deadline-Driven']}
📌 Routine Updates & Check-ins: {category_counts['Routine Updates and Check-Ins']}
📎 Non-Urgent & Informational Emails: {category_counts['Non-Urgent and Informational']}
🎉 Personal & Social Emails: {category_counts['Personal and Social']}
🗑️ Spam/Unimportant Emails Filtered Out: {category_counts['Spam and Unimportant']}

#### AI Conclusion
- Total Critical Emails Requiring Immediate Action: {critical_emails}
- Total Emails That Can Be Deferred or Acknowledged at Convenience: {total_emails - critical_emails}
END CATEGORY COUNTS

Output Requirements:
Return a Markdown summary for with:
    - Total emails
    - Category breakdown
    - List of extracted key entities by category (People, Organizations, Projects, Deadlines)
    - AI Conclusion highlighting urgent issues and action items

*DO NOT estimate or adjust counts; use the provided counts*
*DO NOT enclose the markdown in triple backticks (```) or code blocks.  Output plain markdown only*
"""

# Transform dictionary into JSON object
summary_user_prompt = json.dumps(yesterday_dict, indent=2)

In [13]:

summary_response = llm(summary_system_prompt, summary_user_prompt)
ai_display_markdown(summary_response)

📧 Executive Summary of Emails Received

This summary provides an overview of the emails received by Alex Carter on March 3, 2025. The total number of emails received was 51.

Number of Emails from Yesterday: 51

Breakdown of Emails by Category:

🛑 Urgent & High-Priority Emails: 14

⚡ Deadline-Driven Emails: 4

📌 Routine Updates & Check-ins: 9

📎 Non-Urgent & Informational Emails: 16

🎉 Personal & Social Emails: 4

🗑️ Spam/Unimportant Emails Filtered Out: 4

Key Entities by Category:

Urgent & High-Priority Emails:

- People: David Whitmore, Rachel Thompson, Mark Davidson

- Organizations: Acme Manufacturing, Valve Healthcare

- Projects: Orion Analytics Dashboard, Machine Control System

- Deadlines: Immediate attention required

Deadline-Driven Emails:

- People: Julia Martin, Nathan Cole

- Organizations: Orion Tech Solutions

- Projects: Budget Approval, AWS Resources

- Deadlines: March 7, 2025

Routine Updates & Check-ins:

- People: Samantha Lee, James Patel

- Organizations: Orion Tech Solutions

- Projects: Project Titan, Frontend Updates

- Deadlines: Ongoing

Non-Urgent & Informational Emails:

- People: Lisa Taylor, Mark Davidson

- Organizations: Sentinel Cyber Inc.

- Projects: EBR Meetings, Security AI Module

- Deadlines: March 18, 2025

Personal & Social Emails:

- People: Fiona White, George Harris

- Organizations: Orion Tech Solutions

- Projects: API Security, Supply Chain Security

- Deadlines: None

AI Conclusion

- Total Critical Emails Requiring Immediate Action: 18

- Total Emails That Can Be Deferred or Acknowledged at Convenience: 33

The urgent emails primarily involve system outages and security issues that need immediate resolution to prevent operational disruptions. Action items include addressing the Orion Analytics Dashboard downtime and resolving authentication failures. Non-urgent emails can be reviewed at a later time, focusing on routine updates and informational content.

Urgent and High Priority Summaries¶

In [14]:

urgent_df = yesterday_emails[yesterday_emails['category'] == 'Urgent and High Priority']
urgent_df

Out[14]:

	email_id	date_received	sender	subject	body	main_recipient	email_received	category
4	5	3/3/2025	David Whitmore	[URGENT] Dashboard Syncing Issues – Production...	Hey Alex,\r\n\r\nWe’ve got a big issue right n...	Alex	yesterday	Urgent and High Priority
7	8	3/3/2025	David Kurien	Blocking Issue Alert – Client Data Sync Failing	Hi Alex,\r\n\r\nWe’ve got a major problem—clie...	Alex	yesterday	Urgent and High Priority
9	10	3/3/2025	Tanya Patel	URGENT: Approval Needed for 2-Week Extension o...	CC: John Spencer (PM), Lisa Tan (QA Lead)\r\n\...	Alex	yesterday	Urgent and High Priority
10	11	3/3/2025	David Whitmore	System Crashing During Shift Changes – URGENT	CC: Michael Reynolds (Client Success Manager),...	Alex	yesterday	Urgent and High Priority
13	14	3/3/2025	Bob Smith	?? Security Risk – Critical Patch Delayed	Hi Alex,\r\n\r\nWe have a high-risk delay—the ...	Alex	yesterday	Urgent and High Priority
17	19	3/3/2025	David Whitmore	URGENT: Production Halt – Machine Control Syst...	CC: Michael Reynolds (Client Success Manager)\...	Alex	yesterday	Urgent and High Priority
20	27	3/3/2025	Mark Davidson	[High Priority] Authentication Failing for Mul...	Alex,\r\n\r\nWe have an authentication issue—m...	Alex	yesterday	Urgent and High Priority
23	31	3/3/2025	Mark Davidson	Security Patch Caused System Instability?	Alex,\r\n\r\nAfter the latest security patch, ...	Alex	yesterday	Urgent and High Priority
26	34	3/3/2025	Rachel Lim	URGENT: Approval for Security Audit Vendor – T...	Hi Alex,\r\n\r\nWe still need approval for the...	Alex	yesterday	Urgent and High Priority
28	36	3/3/2025	David Whitmore	URGENT: Critical System Downtime – Immediate A...	Hi Alex,\r\n\r\nWe’re experiencing a major out...	Alex	yesterday	Urgent and High Priority
30	38	3/3/2025	Julia Martin	Escalation – Valve Healthcare (Zoom)	?? Date: March 6, 2025\r\n? Time: 11:00 AM – 1...	Alex	yesterday	Urgent and High Priority
34	42	3/3/2025	Sarah Mitchell	URGENT: Overdue Invoice Approval for Cloud Ser...	Hi Alex,\r\n\r\nThe invoice from CloudOps Solu...	Alex	yesterday	Urgent and High Priority
36	44	3/3/2025	Rachel Thompson	URGENT: Medication Alerts Not Firing – This is...	Alex,\r\n\r\nWe have a serious issue—the autom...	Alex	yesterday	Urgent and High Priority
50	60	3/3/2025	Nathan Ellis	Urgent: Performance Degradation in Production ...	CC: Emma Brooks (Lead Developer), Derek Chan (...	Alex	yesterday	Urgent and High Priority

In [49]:

urgent_system_prompt = f"""You are an automated secretary for a seasoned Senior Manager in Software Development named Alex Carter.  I am going to give you an e-mail.  I need a summary of the emails in a bulleted format using the examples provided below.

{style_guide}

Ensure each email summary follows this format:
  - Subject:
  - Received:
  - Sender Name:
  - Summary:
  - Impact:
  - Next Step: (it should be particular to that email):

BEGIN EXAMPLES
## Urgent and High Priority Summaries
- **Subject: Security Breach Alert - Immediate Response Needed**

  **From:** Diana Wilson (Cybersecurity Lead)

  *Summary:* A critical security vulnerability was discovered in the production environment. Immediate action is required to prevent potential data leaks.


- **Subject: Compliance Issue - Legal Review Needed by EOD**

  **From:** Fiona White (Legal Counsel)

  *Summary:* A third-party software integration has raised compliance concerns. Legal review must be completed today to avoid project delays.


- **Subject: Client Escalation - API Downtime Impacting Operations**

  **From:** Michael Brown (Account Manager)

  *Summary:* A major client has reported API failures affecting their workflow. A resolution and immediate communication are needed.

END EXAMPLES

IMPORTANT: BE SURE TO INCLUDE THE TITLE "Urgent and High Priority Summaries"
"""

# Convert to JSON object
urgent_dict = urgent_df.to_dict(orient='records')
urgent_user_prompt = json.dumps(urgent_dict, indent=2)

In [50]:

urgent_response = llm(urgent_system_prompt, urgent_user_prompt)
ai_display_markdown(urgent_response)

Urgent and High Priority Summaries

1. Subject: [URGENT] Dashboard Syncing Issues – Production Metrics Missing

Received: 3/3/2025

Sender Name: David Whitmore

Summary: Live production metrics are not syncing properly on the Orion Analytics Dashboard, causing data discrepancies and operational challenges.

Impact: Production is running blind, requiring immediate resolution to avoid further operational issues.

Next Step: Confirm if the issue is related to API or data processing lag and provide an update within 24 hours.

2. Subject: Blocking Issue Alert – Client Data Sync Failing

Received: 3/3/2025

Sender Name: David Kurien

Summary: Client transaction data is not syncing for 20% of requests due to timeouts, likely caused by recent deployment changes.

Impact: Clients have noticed missing data, necessitating an urgent action plan.

Next Step: Decide whether to roll back the deployment or isolate the root cause before proceeding.

3. Subject: URGENT: Approval Needed for 2-Week Extension on Acme Corp Deployment

Received: 3/3/2025

Sender Name: Tanya Patel

Summary: A two-week extension is requested for the Acme Corp portal deployment due to internal team PTO conflicts.

Impact: Delaying the deployment will help avoid rushed, low-quality delivery.

Next Step: Decide on approving the new proposed deadline of March 1, 2025.

4. Subject: System Crashing During Shift Changes – URGENT

Received: 3/3/2025

Sender Name: David Whitmore

Summary: The Orion Manufacturing System is crashing during shift changes, causing operational disruptions.

Impact: Operators are unable to log in, leading to lost time and productivity.

Next Step: Join a call at 3 PM EST to discuss the issue further.

5. Subject: ?? Security Risk – Critical Patch Delayed

Received: 3/3/2025

Sender Name: Bob Smith

Summary: A critical security patch rollout is delayed due to dependency conflicts, leaving the system exposed to a known vulnerability.

Impact: Delaying beyond tomorrow risks breaching compliance SLA with Sentinel Cyber Inc.

Next Step: Decide whether to push the patch with risks or delay further.

6. Subject: URGENT: Production Halt – Machine Control System Unresponsive

Received: 3/3/2025

Sender Name: David Whitmore

Summary: The Orion Machine Control System is unresponsive, halting production entirely.

Impact: Full production stop, requiring immediate attention.

Next Step: Call David Whitmore immediately and join the Zoom meeting.

7. Subject: [High Priority] Authentication Failing for Multiple Users

Received: 3/3/2025

Sender Name: Mark Davidson

Summary: Multiple engineers are unable to log in to Orion's security monitoring platform due to an "Invalid session token" error.

Impact: Access issues across teams, requiring urgent resolution.

Next Step: Investigate if this is a known issue or misconfiguration and restore access ASAP.

8. Subject: Security Patch Caused System Instability?

Received: 3/3/2025

Sender Name: Mark Davidson

Summary: After a security patch, automated threat detection accuracy dropped significantly, indicating potential system instability.

Impact: Reduced detection accuracy from 98% to 63%, requiring immediate investigation.

Next Step: Determine if the issue is a configuration problem or something deeper.

9. Subject: URGENT: Approval for Security Audit Vendor – Time-Sensitive

Received: 3/3/2025

Sender Name: Rachel Lim

Summary: Approval is needed for a $25,000 security audit contract with CyberShield, crucial for compliance with a MedTech client.

Impact: Delay may impact compliance certification renewal.

Next Step: Confirm approval by March 7 to proceed with the vendor.

10. Subject: URGENT: Critical System Downtime – Immediate Attention Required

Received: 3/3/2025

Sender Name: David Whitmore

Summary: The Orion Analytics Dashboard is experiencing a major outage, affecting real-time production data access and causing operational delays.

Impact: Dashboard is completely inaccessible, impacting over 50 users.

Next Step: Provide a status update within the next hour and escalate to the engineering team if needed.

11. Subject: Escalation – Valve Healthcare (Zoom)

Received: 3/3/2025

Sender Name: Julia Martin

Summary: ERP System downtime is affecting production, requiring a meeting with Orion Tech Support and Acme IT Team.

Impact: Production is affected, necessitating immediate attention.

Next Step: Attend the Microsoft Teams meeting on March 6, 2025, at 11:00 AM CST.

12. Subject: URGENT: Overdue Invoice Approval for Cloud Services

Received: 3/3/2025

Sender Name: Sarah Mitchell

Summary: An invoice from CloudOps Solutions for $8,500 is overdue by 15 days, risking late fees and potential service disruption.

Impact: Potential penalties and service disruption if not approved promptly.

Next Step: Approve the invoice to avoid penalties and ensure continued service.

13. Subject: URGENT: Medication Alerts Not Firing – This is Dangerous

Received: 3/3/2025

Sender Name: Rachel Thompson

Summary: Automated medication alerts are not firing for ICU patients, posing a significant patient safety risk.

Impact: Immediate risk to patient safety, requiring urgent resolution.

Next Step: Join the emergency call scheduled in 30 minutes.

14. Subject: Urgent: Performance Degradation in Production System

Received: 3/3/2025

Sender Name: Nathan Ellis

Summary: A critical slowdown in the production environment is affecting database response time and client-side API calls.

Impact: Delayed data processing for Acme Manufacturing & Valve Healthcare clients.

Next Step: Approve an emergency scale-up of the database instance and decide on client communication strategy.

Deadline-Driven Summaries¶

In [17]:

deadline_df = yesterday_emails[yesterday_emails['category'] == 'Deadline-Driven']

In [42]:

deadline_system_prompt = f"""You are an automated secretary for a seasoned Senior Manager in Software Development named Alex Carter.  I am going to give you an e-mail.  I need a summary of the emails in a bulleted format using the examples provided below.

{style_guide}

Ensure each email summary follows this format:
  - Subject:
  - Received:
  - Sender Name:
  - Summary:
  - Impact:
  - Deadline:
  - Next Step: (it should be particular to that email):

BEGIN EXAMPLES
## Deadline-Driven Summaries
- **Subject: Security Breach Alert - Immediate Response Needed**

  **From:** Diana Wilson (Cybersecurity Lead)

  *Summary:* A critical security vulnerability was discovered in the production environment. Immediate action is required to prevent potential data leaks.


- **Subject: Compliance Issue - Legal Review Needed by EOD**

  **From:** Fiona White (Legal Counsel)

  *Summary:* A third-party software integration has raised compliance concerns. Legal review must be completed today to avoid project delays.


- **Subject: Client Escalation - API Downtime Impacting Operations**

  **From:** Michael Brown (Account Manager)

  *Summary:* A major client has reported API failures affecting their workflow. A resolution and immediate communication are needed.

END EXAMPLES

IMPORTANT: BE SURE TO INCLUDE THE TITLE "Deadline-Driven Summaries"
"""

# Convert to JSON object
deadline_dict = deadline_df.to_dict(orient='records')
deadline_user_prompt = json.dumps(deadline_dict, indent=2)

In [43]:

deadline_response = llm(deadline_system_prompt, deadline_user_prompt)
ai_display_markdown(deadline_response)

Deadline-Driven Summaries

1. Subject: Approval Request: Budget Approval Needed by EOD

Received: 3/3/2025

Sender Name: Julia Martin

Summary: Budget approval is required by the end of the day to ensure smooth execution of next quarter's projects. Attached is the latest budget breakdown.

Impact: Delays in approval could hinder project timelines.

Deadline: End of today

Next Step: Review the budget breakdown and provide approval or feedback by EOD.

2. Subject: Approval Request: Additional AWS Resources for Project Orion

Received: 3/3/2025

Sender Name: Nathan Cole

Summary: Project Orion requires additional AWS resources to prevent performance bottlenecks. Proposed increase in AWS instance count by 20%.

Impact: Ensures project stays on track for Q2 deadlines.

Deadline: March 7

Next Step: Approve the additional AWS resources by March 7 to avoid service degradation.

3. Subject: Approval Request: Dev Environment Upgrade for Faster Builds

Received: 3/3/2025

Sender Name: Kevin Tran

Summary: Request to upgrade cloud servers to high-performance AWS instances to reduce build times by 60%.

Impact: Faster build times will enhance development efficiency.

Deadline: March 10

Next Step: Approve the server upgrade to deploy by March 10.

4. Subject: Pending Approval – Invoice Dispute (Microsoft Teams)

Received: 3/3/2025

Sender Name: Alice Johnson

Summary: Discussion scheduled to resolve an invoice dispute that has been pending for 14 days.

Impact: Resolution is necessary to maintain financial accuracy and project continuity.

Deadline: March 5, 2025, 4:00 PM – 4:45 PM EST

Next Step: Attend the Microsoft Teams meeting to discuss and resolve the invoice dispute.

First Response Drafts¶

In [51]:

fr_system_prompt = f"""You are an AI secretary for a seasoned Senior Manager in Software Development named Alex Carter.  I am going to give you a list of e-mails in json format.  I need you to create a draft response for each email in the list.  I will provide you with the following examples as a template.  Ensure each draft response follows this format, acknowledges the sender's request, is contextually relevant, addresses key points or actions, provides clear next steps, and aligns with a professional and corporate manner of communication:

{style_guide}

*BEGIN EXAMPLES*
## First Response Drafts

Here are the AI-drafted responses to the critical emails received yesterday:
**Subject:** Budget Approval Needed by EOD
**Sender Name:** Julia Martin
**AI Drafted Reply:**
Dear Julia,

Thank you for your email and for providing the budget breakdown.  I will review the attached document and ensure that my approval is provided by the end of the day to facilitate the smooth execution of next quarter's projects.

Best regards,
Alex

---

**Subject:** Security Breach Alert - Immediate Response Needed
**Sender Name:** Ian Adams
**AI Drafted Reply:**
Dear Ian,

Thank you for bringing the security breach to my attention.  I will review the details immediately and provide my thoughts as soon as possible.

Best regards,
Alex

---

**Subject:** Compliance Issue - Legal Review Needed by EOD
**Sender Name:** Diana Wilson
**AI Drafted Reply:**
Dear Diana,

Thank you for your email regarding the compliance issue.  I will review the details and provide my feedback by the end of the day as requested.

Best regards,
Alex
*END EXAMPLES*

IMPORTANT: BE SURE TO INCLUDE THE TITLE "First Response Drafts"
"""

fr_emails = yesterday_emails[yesterday_emails['category'].isin(['Urgent and High Priority', 'Deadline-Driven'])]

# Convert to JSON
fr_dict = fr_emails.to_dict(orient='records')
fr_user_prompt = json.dumps(fr_dict, indent=2)

In [52]:

fr_response = llm(fr_system_prompt, fr_user_prompt)
ai_display_markdown(fr_response)

First Response Drafts

Subject: Approval Request: Budget Approval Needed by EOD

Dear Julia,

Thank you for your email and for providing the budget breakdown. I will review the attached document and ensure that my approval is provided by the end of the day to facilitate the smooth execution of next quarter's projects.

Best regards,
Alex

Subject: [URGENT] Dashboard Syncing Issues – Production Metrics Missing

Dear David,

Thank you for alerting me to the dashboard syncing issues. I will have my team investigate the potential API or data processing lag immediately. We will aim to provide you with an update within the next 24 hours. Please let me know if you require any further assistance from our IT side.

Best regards,
Alex

Subject: Approval Request: Additional AWS Resources for Project Orion

Dear Nathan,

Thank you for bringing the AWS resource requirements to my attention. I will review the proposed solution and aim to provide my approval by March 7 to ensure we avoid any service degradation and stay on track for Q2 deadlines.

Best regards,
Alex

Subject: Blocking Issue Alert – Client Data Sync Failing

Dear David,

Thank you for notifying me about the client data sync issue. I will coordinate with the team to determine whether a rollback is necessary or if we can isolate the root cause quickly. I will keep you updated on our progress.

Best regards,
Alex

Subject: URGENT: Approval Needed for 2-Week Extension on Acme Corp Deployment

Dear Tanya,

Thank you for informing me about the need for a 2-week extension on the Acme Corp deployment. I understand the circumstances and will review the situation to provide a decision promptly. Please ensure that all necessary steps are taken to avoid further delays.

Best regards,
Alex

Subject: System Crashing During Shift Changes – URGENT

Dear David,

Thank you for bringing the system crashing issue to my attention. I will have someone from my team investigate the server load or authentication issues immediately. I will join the call at 3 PM EST to discuss further.

Best regards,
Alex

Subject: Security Risk – Critical Patch Delayed

Dear Bob,

Thank you for informing me about the delay in the security patch rollout. I will review the current status and discuss with the team whether to proceed with the patch or delay further. I will get back to you with a decision shortly.

Best regards,
Alex

Subject: URGENT: Production Halt – Machine Control System Unresponsive

Dear David,

Thank you for alerting me to the unresponsive machine control system. I will have someone from my team contact you immediately to address this critical issue. I will join the Zoom call to discuss further actions.

Best regards,
Alex

Subject: [High Priority] Authentication Failing for Multiple Users

Dear Mark,

Thank you for bringing the authentication issue to my attention. I will have my team investigate the "Invalid session token" error immediately and work to restore access as soon as possible.

Best regards,
Alex

Subject: Security Patch Caused System Instability?

Dear Mark,

Thank you for notifying me about the system instability following the security patch. I will have someone from my team look into the configuration or deeper issues affecting the detection accuracy.

Best regards,
Alex

Subject: URGENT: Approval for Security Audit Vendor – Time-Sensitive

Dear Rachel,

Thank you for the reminder regarding the security audit contract with CyberShield. I will review the contract details and aim to provide my approval by March 7 to ensure compliance certification renewal is not impacted.

Best regards,
Alex

Subject: URGENT: Critical System Downtime – Immediate Attention Required

Dear David,

Thank you for informing me about the major outage with the Orion Analytics Dashboard. I will escalate this to the engineering team immediately to find a resolution or workaround by 12 PM EST today. I will provide a status update within the next hour.

Best regards,
Alex

Subject: Escalation – Valve Healthcare (Zoom)

Dear Julia,

Thank you for the meeting invitation regarding the ERP system downtime. I will ensure that Orion Tech Support is prepared to address the issue during the meeting on March 6 at 11:00 AM CST.

Best regards,
Alex

Subject: Approval Request: Dev Environment Upgrade for Faster Builds

Dear Kevin,

Thank you for the request to upgrade the development environment. I will review the proposed upgrade and aim to provide my approval by March 10 to facilitate faster build times.

Best regards,
Alex

Subject: URGENT: Overdue Invoice Approval for Cloud Services

Dear Sarah,

Thank you for bringing the overdue invoice to my attention. I will review the details and aim to provide my approval as soon as possible to avoid any late fees or service disruptions.

Best regards,
Alex

Subject: URGENT: Medication Alerts Not Firing – This is Dangerous

Dear Rachel,

Thank you for alerting me to the issue with the medication alerts. I will ensure that our team addresses this patient safety risk immediately. I will join the emergency call in 30 minutes to discuss further actions.

Best regards,
Alex

Subject: Pending Approval – Invoice Dispute (Microsoft Teams)

Dear Alice,

Thank you for the meeting invitation regarding the invoice dispute. I will ensure that I am available for the discussion on March 5 at 4:00 PM EST to resolve the issue promptly.

Best regards,
Alex

Subject: Urgent: Performance Degradation in Production System

Dear Nathan,

Thank you for notifying me about the performance degradation in the production system. I will review the situation and approve the emergency scale-up of the database instance. Let's discuss the client communication strategy during the Google Meet at 2:30 PM EST.

Best regards,
Alex

Evaluation¶

LLM as a Judge¶

In [22]:

eval_system_prompt = """
Rate the given **summary** across the following criteria on a scale of 1 to 5:
- **Relevance:** How well the summary addresses the input query or task. Highlight which key information is captured or missed.
- **Clarity:** How clear and understandable the e-mail summaries are. Mention any confusing or ambiguous phrases.
- **Actionability:** Whether the e-mail summaries provides clear next steps or actionable information.

Provide your evaluation strictly in JSON format:
{
  "Relevance": {"score": score, "justification": "reason + example"},
  "Clarity": {"score": score, "justification": "reason + example"},
  "Actionability": {"score": score, "justification": "reason + example"},
  "Strengths": "Highlight key strengths of the summary",
  "Improvements": "Suggest 1-2 improvements",
  "Overall_Justification": "Summary evaluation in 2-3 lines with key observations"
}"""

In [23]:

# Get LLM as a Judge evaluation
def evaluate_summary(summary, eval_model="gpt-4-turbo"):
    user_prompt = f"""

    <Your user prompt content>

    Here is the **summary**: {summary}
    """




    try:
        eval_response = client.chat.completions.create(
            model=eval_model,
            messages=[
                {"role": "system", "content": eval_system_prompt},
                {"role": "user", "content": user_prompt}
            ],
            temperature=0
        )
        return eval_response.choices[0].message.content.strip()

    except Exception as e:
        print(f"Error evaluating prompt: {e}")
        return "{}"  # Return empty JSON structure on error

In [24]:

responses = [summary_response, urgent_response, deadline_response, fr_response]
evaluation_results = [evaluate_summary(response) for response in responses]

In [25]:

scores = []
justifications = []
strengths = []
improvements = []


for result in evaluation_results:
    try:
        result_dict = json.loads(result)
        score_dict = {k: v.get("score", "NA") for k, v in result_dict.items() if isinstance(v, dict) and "score" in v}
        justification = result_dict.get("Overall_Justification", "NA")
        strength = result_dict.get("Strengths", "NA")
        improvement = result_dict.get("Improvements", "NA")

        # Append Results
        scores.append(score_dict)
        justifications.append(justification)
        strengths.append(strength)
        improvements.append(improvement)

    except (json.JSONDecodeError, KeyError, TypeError):
        scores.append({"Relevance": None, "Clarity": None, "Conciseness": None, "Coherence": None, "Actionability": None})
        justifications.append("")
        strengths.append("")
        improvements.append("")

In [26]:

# Creating Final Evaluation DataFrame
df_scores = pd.DataFrame(scores, index=['summary', 'urgent', 'deadline', 'first response'])
df_scores["Strengths"] = strengths
df_scores["Improvements"] = improvements
df_scores["Justification"] = justifications

In [27]:

df_scores

Out[27]:

	Relevance	Clarity	Actionability	Strengths	Improvements	Justification
summary	5	5	5	The summary is comprehensive, well-organized, ...	Include a direct recommendation for the order ...	The summary is highly relevant, clear, and act...
urgent	5	5	5	The summary is comprehensive and well-organize...	Include a brief status or urgency level for ea...	The summary is highly relevant, clear, and act...
deadline	5	5	5	The summary is well-structured, using color an...	Include a brief mention of the consequences of...	The summary is highly relevant, clear, and act...
first response	5	5	5	The summaries are well-structured and directly...	Include specific timelines or deadlines for th...	The summaries are highly relevant, clear, and ...

In [28]:

for index, row in df_scores.iterrows():
    print(index.title())
    print(f"""\tRelevance: {row['Relevance']}
    Clarity: {row['Clarity']}
    Actionability: {row['Actionability']}

    Strengths: {row['Strengths']}

    Improvements: {row['Improvements']}

    Justification: {row['Justification']}

    """)

Summary
Relevance: 5
Clarity: 5
Actionability: 5

Strengths: The summary is comprehensive, well-organized, and visually distinct, making it easy to navigate. It effectively categorizes emails and provides specific details about the people, organizations, projects, and deadlines involved.

Improvements: Include a direct recommendation for the order of email prioritization based on urgency and importance. Provide a brief suggestion on how to handle non-urgent emails more efficiently.

Justification: The summary is highly relevant, clear, and actionable. It provides a detailed breakdown of emails, making it easy for the recipient to understand the day's priorities and plan accordingly.

Urgent
Relevance: 5
Clarity: 5
Actionability: 5

Strengths: The summary is comprehensive and well-organized, covering all critical aspects of the emails. It uses a consistent format that makes it easy to scan through and identify key actions.

Improvements: Include a brief status or urgency level for each issue to prioritize actions further. Consider summarizing the potential consequences if the next steps are not taken to highlight the importance of timely actions.

Justification: The summary is highly relevant, clear, and actionable, effectively condensing urgent and high-priority issues into a format that facilitates quick understanding and decision-making.

Deadline
Relevance: 5
Clarity: 5
Actionability: 5

Strengths: The summary is well-structured, using color and formatting to enhance readability and quick scanning. It succinctly captures the essence of each email, focusing on actionable items and deadlines.

Improvements: Include a brief mention of the consequences of not taking the recommended actions to emphasize the importance of the deadlines and actions.

Justification: The summary is highly relevant, clear, and actionable. It effectively condenses the key information from multiple emails into a digestible format, making it easy for the reader to understand and act upon.

First Response
Relevance: 5
Clarity: 5
Actionability: 5

Strengths: The summaries are well-structured and directly address the recipients, making them personal and engaging. They effectively communicate the necessary actions and responses in a professional tone.

Improvements: Include specific timelines or deadlines for the actions when not mentioned to enhance clarity on expected completion times.

Justification: The summaries are highly relevant, clear, and actionable, effectively condensing the main points of the emails into concise responses that facilitate quick understanding and follow-up.

Categorization Accuracy¶

Base Categories¶

In [29]:

true_categories = df_true[['email_id', 'true_category']]
comparison_df = pd.merge(yesterday_emails, true_categories, left_on='email_id', right_on='email_id', how='left')

In [30]:

from sklearn.metrics import confusion_matrix, classification_report
import matplotlib.pyplot as plt

In [31]:

# Create confusion matrix
labels = [
    'Urgent and High Priority',
    'Deadline-Driven',
    'Routine Updates and Check-Ins',
    'Non-Urgent and Informational',
    'Personal and Social',
    'Spam and Unimportant'
]

cm = confusion_matrix(comparison_df['true_category'], comparison_df['category'], labels=labels)

# Plot confusion matrix
plt.figure(figsize=(10,10))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues',
            xticklabels=labels, yticklabels=labels)
plt.xlabel('AI Prediction')
plt.ylabel('Human Categorization')
plt.title('E-Mail Categorization Confusion Matrix')

# Rotate xticks 45° and center-align
plt.xticks(rotation=45, ha='right', rotation_mode='anchor')
plt.yticks(rotation=0, va='center')

plt.show()

# Generate classification metrics report
report = classification_report(comparison_df['true_category'], comparison_df['category'], zero_division=0)
print(report)

                               precision    recall  f1-score   support

              Deadline-Driven       0.50      0.50      0.50         4
 Non-Urgent and Informational       0.81      0.87      0.84        15
          Personal and Social       1.00      0.80      0.89         5
Routine Updates and Check-Ins       0.67      0.67      0.67         9
         Spam and Unimportant       1.00      1.00      1.00         4
     Urgent and High Priority       0.79      0.79      0.79        14

                     accuracy                           0.78        51
                    macro avg       0.79      0.77      0.78        51
                 weighted avg       0.79      0.78      0.78        51

Critical vs. Non-Critical Accuracy¶

In [32]:

# Map categories to critical/non-critical
comparison_mapping = {
    'Urgent and High Priority': 'Critical',
    'Deadline-Driven': 'Critical',
    'Routine Updates and Check-Ins': 'Non-Critical',
    'Non-Urgent and Informational': 'Non-Critical',
    'Personal and Social': 'Non-Critical',
    'Spam and Unimportant': 'Non-Critical'
}
comparison_df['critical_pred'] = comparison_df['category'].map(comparison_mapping)
comparison_df['critical_true'] = comparison_df['true_category'].map(comparison_mapping)


# Create confusion matrix
labels = ['Critical', 'Non-Critical']

critical_cm = confusion_matrix(comparison_df['critical_true'], comparison_df['critical_pred'], labels=labels)

# Plot confusion matrix
plt.figure(figsize=(8,8))
sns.heatmap(critical_cm, annot=True, fmt='d', cmap='Blues',
            xticklabels=labels, yticklabels=labels)
plt.xlabel('AI Prediction')
plt.ylabel('Human Categorization')
plt.title('Critical E-Mail Confusion Matrix')

# Rotate xticks 45° and center-align
plt.xticks(rotation=45, ha='right', rotation_mode='anchor')
plt.yticks(rotation=0, va='center')

plt.show()

# Generate classification metrics report
critical_report = classification_report(comparison_df['critical_true'], comparison_df['critical_pred'], zero_division=0)
print(critical_report)

              precision    recall  f1-score   support

    Critical       0.89      0.89      0.89        18
Non-Critical       0.94      0.94      0.94        33

    accuracy                           0.92        51
   macro avg       0.91      0.91      0.91        51
weighted avg       0.92      0.92      0.92        51

Full Output¶

In [53]:

# Print the full output as it would be seen in live production
summaries = [
    summary_response,
    urgent_response,
    deadline_response,
    fr_response
]
blue_line = '<hr style="border: none; height: 3px; background-color: #2196F3; margin: 20px 0;">'
neon_line = '<hr style="border: none; height: 4px; background: linear-gradient(to right, #00c6ff, #0072ff); box-shadow: 0 0 10px #00c6ff;">'

all_summaries = f"\n\n\n{neon_line}\n\n\n".join(summaries)
ai_display_markdown(all_summaries)

📧 Executive Summary of Emails Received

This summary provides an overview of the emails received by Alex Carter on March 3, 2025. The total number of emails received was 51.

Number of Emails from Yesterday: 51

Breakdown of Emails by Category:

🛑 Urgent & High-Priority Emails: 14

⚡ Deadline-Driven Emails: 4

📌 Routine Updates & Check-ins: 9

📎 Non-Urgent & Informational Emails: 16

🎉 Personal & Social Emails: 4

🗑️ Spam/Unimportant Emails Filtered Out: 4

Key Entities by Category:

Urgent & High-Priority Emails:

- People: David Whitmore, Rachel Thompson, Mark Davidson

- Organizations: Acme Manufacturing, Valve Healthcare

- Projects: Orion Analytics Dashboard, Machine Control System

- Deadlines: Immediate attention required

Deadline-Driven Emails:

- People: Julia Martin, Nathan Cole

- Organizations: Orion Tech Solutions

- Projects: Budget Approval, AWS Resources

- Deadlines: March 7, 2025

Routine Updates & Check-ins:

- People: Samantha Lee, James Patel

- Organizations: Orion Tech Solutions

- Projects: Project Titan, Frontend Updates

- Deadlines: Ongoing

Non-Urgent & Informational Emails:

- People: Lisa Taylor, Mark Davidson

- Organizations: Sentinel Cyber Inc.

- Projects: EBR Meetings, Security AI Module

- Deadlines: March 18, 2025

Personal & Social Emails:

- People: Fiona White, George Harris

- Organizations: Orion Tech Solutions

- Projects: API Security, Supply Chain Security

- Deadlines: None

AI Conclusion

- Total Critical Emails Requiring Immediate Action: 18

- Total Emails That Can Be Deferred or Acknowledged at Convenience: 33

Urgent and High Priority Summaries

1. Subject: [URGENT] Dashboard Syncing Issues – Production Metrics Missing

Received: 3/3/2025

Sender Name: David Whitmore

Summary: Live production metrics are not syncing properly on the Orion Analytics Dashboard, causing data discrepancies and operational challenges.

Impact: Production is running blind, requiring immediate resolution to avoid further operational issues.

Next Step: Confirm if the issue is related to API or data processing lag and provide an update within 24 hours.

2. Subject: Blocking Issue Alert – Client Data Sync Failing

Received: 3/3/2025

Sender Name: David Kurien

Summary: Client transaction data is not syncing for 20% of requests due to timeouts, likely caused by recent deployment changes.

Impact: Clients have noticed missing data, necessitating an urgent action plan.

Next Step: Decide whether to roll back the deployment or isolate the root cause before proceeding.

3. Subject: URGENT: Approval Needed for 2-Week Extension on Acme Corp Deployment

Received: 3/3/2025

Sender Name: Tanya Patel

Summary: A two-week extension is requested for the Acme Corp portal deployment due to internal team PTO conflicts.

Impact: Delaying the deployment will help avoid rushed, low-quality delivery.

Next Step: Decide on approving the new proposed deadline of March 1, 2025.

4. Subject: System Crashing During Shift Changes – URGENT

Received: 3/3/2025

Sender Name: David Whitmore

Summary: The Orion Manufacturing System is crashing during shift changes, causing operational disruptions.

Impact: Operators are unable to log in, leading to lost time and productivity.

Next Step: Join a call at 3 PM EST to discuss the issue further.

5. Subject: ?? Security Risk – Critical Patch Delayed

Received: 3/3/2025

Sender Name: Bob Smith

Summary: A critical security patch rollout is delayed due to dependency conflicts, leaving the system exposed to a known vulnerability.

Impact: Delaying beyond tomorrow risks breaching compliance SLA with Sentinel Cyber Inc.

Next Step: Decide whether to push the patch with risks or delay further.

6. Subject: URGENT: Production Halt – Machine Control System Unresponsive

Received: 3/3/2025

Sender Name: David Whitmore

Summary: The Orion Machine Control System is unresponsive, halting production entirely.

Impact: Full production stop, requiring immediate attention.

Next Step: Call David Whitmore immediately and join the Zoom meeting.

7. Subject: [High Priority] Authentication Failing for Multiple Users

Received: 3/3/2025

Sender Name: Mark Davidson

Summary: Multiple engineers are unable to log in to Orion's security monitoring platform due to an "Invalid session token" error.

Impact: Access issues across teams, requiring urgent resolution.

Next Step: Investigate if this is a known issue or misconfiguration and restore access ASAP.

8. Subject: Security Patch Caused System Instability?

Received: 3/3/2025

Sender Name: Mark Davidson

Summary: After a security patch, automated threat detection accuracy dropped significantly, indicating potential system instability.

Impact: Reduced detection accuracy from 98% to 63%, requiring immediate investigation.

Next Step: Determine if the issue is a configuration problem or something deeper.

9. Subject: URGENT: Approval for Security Audit Vendor – Time-Sensitive

Received: 3/3/2025

Sender Name: Rachel Lim

Summary: Approval is needed for a $25,000 security audit contract with CyberShield, crucial for compliance with a MedTech client.

Impact: Delay may impact compliance certification renewal.

Next Step: Confirm approval by March 7 to proceed with the vendor.

10. Subject: URGENT: Critical System Downtime – Immediate Attention Required

Received: 3/3/2025

Sender Name: David Whitmore

Summary: The Orion Analytics Dashboard is experiencing a major outage, affecting real-time production data access and causing operational delays.

Impact: Dashboard is completely inaccessible, impacting over 50 users.

Next Step: Provide a status update within the next hour and escalate to the engineering team if needed.

11. Subject: Escalation – Valve Healthcare (Zoom)

Received: 3/3/2025

Sender Name: Julia Martin

Summary: ERP System downtime is affecting production, requiring a meeting with Orion Tech Support and Acme IT Team.

Impact: Production is affected, necessitating immediate attention.

Next Step: Attend the Microsoft Teams meeting on March 6, 2025, at 11:00 AM CST.

12. Subject: URGENT: Overdue Invoice Approval for Cloud Services

Received: 3/3/2025

Sender Name: Sarah Mitchell

Summary: An invoice from CloudOps Solutions for $8,500 is overdue by 15 days, risking late fees and potential service disruption.

Impact: Potential penalties and service disruption if not approved promptly.

Next Step: Approve the invoice to avoid penalties and ensure continued service.

13. Subject: URGENT: Medication Alerts Not Firing – This is Dangerous

Received: 3/3/2025

Sender Name: Rachel Thompson

Summary: Automated medication alerts are not firing for ICU patients, posing a significant patient safety risk.

Impact: Immediate risk to patient safety, requiring urgent resolution.

Next Step: Join the emergency call scheduled in 30 minutes.

14. Subject: Urgent: Performance Degradation in Production System

Received: 3/3/2025

Sender Name: Nathan Ellis

Summary: A critical slowdown in the production environment is affecting database response time and client-side API calls.

Impact: Delayed data processing for Acme Manufacturing & Valve Healthcare clients.

Next Step: Approve an emergency scale-up of the database instance and decide on client communication strategy.

Deadline-Driven Summaries

1. Subject: Approval Request: Budget Approval Needed by EOD

Received: 3/3/2025

Sender Name: Julia Martin

Summary: Budget approval is required by the end of the day to ensure smooth execution of next quarter's projects. Attached is the latest budget breakdown.

Impact: Delays in approval could hinder project timelines.

Deadline: End of today

Next Step: Review the budget breakdown and provide approval or feedback by EOD.

2. Subject: Approval Request: Additional AWS Resources for Project Orion

Received: 3/3/2025

Sender Name: Nathan Cole

Summary: Project Orion requires additional AWS resources to prevent performance bottlenecks. Proposed increase in AWS instance count by 20%.

Impact: Ensures project stays on track for Q2 deadlines.

Deadline: March 7

Next Step: Approve the additional AWS resources by March 7 to avoid service degradation.

3. Subject: Approval Request: Dev Environment Upgrade for Faster Builds

Received: 3/3/2025

Sender Name: Kevin Tran

Summary: Request to upgrade cloud servers to high-performance AWS instances to reduce build times by 60%.

Impact: Faster build times will enhance development efficiency.

Deadline: March 10

Next Step: Approve the server upgrade to deploy by March 10.

4. Subject: Pending Approval – Invoice Dispute (Microsoft Teams)

Received: 3/3/2025

Sender Name: Alice Johnson

Summary: Discussion scheduled to resolve an invoice dispute that has been pending for 14 days.

Impact: Resolution is necessary to maintain financial accuracy and project continuity.

Deadline: March 5, 2025, 4:00 PM – 4:45 PM EST

Next Step: Attend the Microsoft Teams meeting to discuss and resolve the invoice dispute.

First Response Drafts

Subject: Approval Request: Budget Approval Needed by EOD

Dear Julia,

Best regards,
Alex

Subject: [URGENT] Dashboard Syncing Issues – Production Metrics Missing

Dear David,

Best regards,
Alex

Subject: Approval Request: Additional AWS Resources for Project Orion

Dear Nathan,

Best regards,
Alex

Subject: Blocking Issue Alert – Client Data Sync Failing

Dear David,

Best regards,
Alex

Subject: URGENT: Approval Needed for 2-Week Extension on Acme Corp Deployment

Dear Tanya,

Best regards,
Alex

Subject: System Crashing During Shift Changes – URGENT

Dear David,

Best regards,
Alex

Subject: Security Risk – Critical Patch Delayed

Dear Bob,

Best regards,
Alex

Subject: URGENT: Production Halt – Machine Control System Unresponsive

Dear David,

Best regards,
Alex

Subject: [High Priority] Authentication Failing for Multiple Users

Dear Mark,

Thank you for bringing the authentication issue to my attention. I will have my team investigate the "Invalid session token" error immediately and work to restore access as soon as possible.

Best regards,
Alex

Subject: Security Patch Caused System Instability?

Dear Mark,

Thank you for notifying me about the system instability following the security patch. I will have someone from my team look into the configuration or deeper issues affecting the detection accuracy.

Best regards,
Alex

Subject: URGENT: Approval for Security Audit Vendor – Time-Sensitive

Dear Rachel,

Best regards,
Alex

Subject: URGENT: Critical System Downtime – Immediate Attention Required

Dear David,

Best regards,
Alex

Subject: Escalation – Valve Healthcare (Zoom)

Dear Julia,

Thank you for the meeting invitation regarding the ERP system downtime. I will ensure that Orion Tech Support is prepared to address the issue during the meeting on March 6 at 11:00 AM CST.

Best regards,
Alex

Subject: Approval Request: Dev Environment Upgrade for Faster Builds

Dear Kevin,

Thank you for the request to upgrade the development environment. I will review the proposed upgrade and aim to provide my approval by March 10 to facilitate faster build times.

Best regards,
Alex

Subject: URGENT: Overdue Invoice Approval for Cloud Services

Dear Sarah,

Thank you for bringing the overdue invoice to my attention. I will review the details and aim to provide my approval as soon as possible to avoid any late fees or service disruptions.

Best regards,
Alex

Subject: URGENT: Medication Alerts Not Firing – This is Dangerous

Dear Rachel,

Best regards,
Alex

Subject: Pending Approval – Invoice Dispute (Microsoft Teams)

Dear Alice,

Thank you for the meeting invitation regarding the invoice dispute. I will ensure that I am available for the discussion on March 5 at 4:00 PM EST to resolve the issue promptly.

Best regards,
Alex

Subject: Urgent: Performance Degradation in Production System

Dear Nathan,

Best regards,
Alex

Summary and Recommendation¶

Summary of Observations¶

The AI E-mail Secretary performed surprisingly well in both summarizing e-mails and extracting key dates/names, as well as intelligently highlighting key observations contained in the e-mails. The generated e-mail responses were also robust and were pertinent to the requests or deadlines contained in the received e-mails. One issue observed is that categorization versus human input was less accurate than would be anticipated, but this is largely due to the small sample size of 51 e-mails from the previous day. A larger sample size would most likely show improved accuracy, recall, precision, and F1 scores. As well, there were several e-mails that could be categorized into either the "Deadline-Driven" or "Urgent and High Priority" categories and the business rules surrounding whether to categorize e-mails where Alex is not the main recipient into one category or spread amongst several based on priority was also an issue and is largely dependent on human interpretation.

Evaluation Highlights¶

In [54]:

# Split e-mails based on the line delimiter provided through LLM output styling
delimiter = '<hr style="border:none;height:2px;background:linear-gradient(to right,#00FF7F,#39FF14);box-shadow:0 0 10px #39FF14;margin:20px 0;">'

email_summaries = urgent_response.split(delimiter)
clean_summaries = [re.sub(r'<[^>]+>', '', s) for s in email_summaries]

#Drop the title
clean_summaries = clean_summaries[1:]

comparison_df = pd.DataFrame({
    "Original Email": urgent_df['body'].values,
    "Summary": [s.strip().replace('**', '') for s in clean_summaries if s.strip()]
})

# Select 5 random e-mails
random_samples = comparison_df.sample(n=5, random_state=42)

# Show original e-mail and summary side-by-side
for index, row in random_samples.iterrows():
    display(Markdown("##### Original Email: "))
    display(HTML(f"<span style='color:#aa7dff; font-style:italic;'>{row['Original Email'].replace('\n', '<br>')}</span><br><br>"))
    display(Markdown("\n\n##### Summary: "))
    display(HTML(f"<span style='color:#64aafa;'>{row['Summary'].replace('\n', '<br>')}</span>"))
    print("\n" + "-"*80 + "\n")

Original Email:¶

Hi Alex,

We’re experiencing a major outage with the Orion Analytics Dashboard, and our team is unable to access real-time production data. This issue is impacting multiple manufacturing lines, causing delays in operations.

Issue Details:
?? Impact: Dashboard is completely inaccessible (error 500)
?? Start Time: March 5, 2025, 08:15 AM EST
?? Affected Users: Entire production team (50+ users)
?? Attempted Fixes: Cleared cache, restarted local systems – no change

Urgent Request:
We need an immediate resolution or a workaround by 12 PM EST today. If this is a backend issue, please escalate to the engineering team ASAP.

Please provide a status update within the next hour. If required, we are available for a call at your earliest convenience.

Best,
David Whitmore
IT Operations Manager, Acme Manufacturing

Summary:¶

10. Subject: URGENT: Critical System Downtime – Immediate Attention Required
Received: 3/3/2025
Sender Name: David Whitmore
Summary: The Orion Analytics Dashboard is experiencing a major outage, affecting real-time production data access and causing operational delays.
Impact: Dashboard is completely inaccessible, impacting over 50 users.
Next Step: Provide a status update within the next hour and escalate to the engineering team if needed.

--------------------------------------------------------------------------------

Original Email:¶

Hi Alex,

The invoice from CloudOps Solutions for $8,500 is now overdue by 15 days.

?? Service Provided: Cloud security monitoring
?? Original Due Date: February 20, 2025
?? Risk: Late fees + potential service disruption

Can we get this approved ASAP to avoid penalties?

Best,
Sarah Mitchell
Finance Lead, Orion Tech Solutions

Summary:¶

12. Subject: URGENT: Overdue Invoice Approval for Cloud Services
Received: 3/3/2025
Sender Name: Sarah Mitchell
Summary: An invoice from CloudOps Solutions for $8,500 is overdue by 15 days, risking late fees and potential service disruption.
Impact: Potential penalties and service disruption if not approved promptly.
Next Step: Approve the invoice to avoid penalties and ensure continued service.

--------------------------------------------------------------------------------

Original Email:¶

Hey Alex,

We’ve got a big issue right now—live production metrics aren’t syncing properly on the Orion Analytics Dashboard. Some numbers are missing, others just look wrong. We’re seeing data gaps in yesterday’s batch reports, and my team can’t verify production outputs.

Couple of things:

This started sometime overnight. No one touched the system, but by 7 AM today, we noticed discrepancies.
Not clear if this is an API issue or data processing lag. Can your team confirm?
Production is running blind right now—we need this resolved ASAP or at least tell us how to get accurate numbers another way.
Need an update in the next 24 hours, or I’ll have to escalate further. If you need someone from our IT side, let me know.

Thanks,
David Whitmore
VP, IT Operations
Acme Manufacturing

Summary:¶

1. Subject: [URGENT] Dashboard Syncing Issues – Production Metrics Missing
Received: 3/3/2025
Sender Name: David Whitmore
Summary: Live production metrics are not syncing properly on the Orion Analytics Dashboard, causing data discrepancies and operational challenges.
Impact: Production is running blind, requiring immediate resolution to avoid further operational issues.
Next Step: Confirm if the issue is related to API or data processing lag and provide an update within 24 hours.

--------------------------------------------------------------------------------

Original Email:¶

Alex,

We have a serious issue—the automated medication alerts are not firing for ICU patients. This is a patient safety risk.

This needs to be fixed immediately. Sending an invite for an emergency call in 30 mins.

Rachel Thompson
AVP, Clinical Systems, Valve healthcare

Summary:¶

13. Subject: URGENT: Medication Alerts Not Firing – This is Dangerous
Received: 3/3/2025
Sender Name: Rachel Thompson
Summary: Automated medication alerts are not firing for ICU patients, posing a significant patient safety risk.
Impact: Immediate risk to patient safety, requiring urgent resolution.
Next Step: Join the emergency call scheduled in 30 minutes.

--------------------------------------------------------------------------------

Original Email:¶

CC: Michael Reynolds (Client Success Manager)

Alex,

We have a critical issue—the Orion Machine Control System is not responding. Line operators are unable to send commands to machinery, and production is effectively at a standstill.

Started: March 5, 2025, 7:45 AM
Systems Affected: Orion Control Module (all units)
Impact: Full production stop

This needs immediate attention. Can someone call me right now? (Adding a Zoom invite)

David Whitmore (VP, IT Operations)
ACME MANUFACTURING

Summary:¶

6. Subject: URGENT: Production Halt – Machine Control System Unresponsive
Received: 3/3/2025
Sender Name: David Whitmore
Summary: The Orion Machine Control System is unresponsive, halting production entirely.
Impact: Full production stop, requiring immediate attention.
Next Step: Call David Whitmore immediately and join the Zoom meeting.

--------------------------------------------------------------------------------

Relevance¶

As shown above, each summary was relevant to the pertinent e-mail and included almost all important points, along with next steps.

Clarity¶

The summary was easy to understand, but could be condensed further. For instance, the e-mail pertaining to the automated medication alerts was repeated almost verbatim in the summary due to its short length. The AI assistant could be further tasked with condensing the relevant information with shorter paraphrasing.

Actionability¶

The AI assistant accurately pinpointed next steps for each e-mail, including appropriate timeframes for responses.

Strengths¶

The AI assistant excelled at capturing deadlines, named entities, and providing next steps based on the e-mails. It was particularly strong at summarizing key points for the urgent and high priority e-mails so that they could be acted upon expediently. Summaries also omitted extraneous information that, while relevant, was not necessary for the summarization. Categorization was lacking accuracy across the 6 main categories, but when grouped by "Critical" and "Non-Critical", the accuracy greatly improved.

Improvement Areas¶

While the AI assistant summarized e-mails well, the initial categorization had lower accuracy than anticipated, particularly regarding "Urgent and High Priority" and "Deadline-Driven" emails. This may be largely due to the fact that "Deadline-Driven" e-mails are often "Urgent and High Priority", such as when an e-mail states that something needs to be addressed by a certain time, typically in the near future. As well, "Routine Updates and Check-Ins" sometimes had deadlines or mentioned urgency, but the entirety of the e-mail would be classified as being a check-in or update. It is these overlapping categorizations that need to be addressed in the model.

Final Recommendation¶

The AI Assistant, while slightly inaccurate at categorization, shows high promise in a production environment. It accurately summarizes e-mails, extracts relevant information like named entities and deadlines, and provides specific action items for each e-mail. Improvements need to be made in the base categorization of e-mails to ensure that e-mails are placed in the correct categories, but even mis-categorized e-mails still are largely represented in the "Critical" and "Non-Critical" breakouts of the executive summary. This ensures that even if an e-mail is mis-categorized, it is still placed in the correct priority bucket with a high degree of accuracy and the user will still apply the appropriate urgency to the e-mail. The categorization of e-mails could be improved with further prompt refinement and more rigorous methodologies as well as by implementing RAG-based methodologies to further enhance the accuracy of the model. With a few minor improvements in the categorization of e-mails and more condensed summarization, the AI Assistant could be a robust addition to any e-mail inbox.

Mike Anderson

AI E-Mail Assistant

The Case Study¶

Data Cleaning¶

LLM Output Styling Rules¶

Categorization of E-Mails¶

Executive E-Mail Summary¶

📧 Executive Summary of Emails Received

Number of Emails from Yesterday: 51

Breakdown of Emails by Category:

Key Entities by Category:

AI Conclusion

Urgent and High Priority Summaries¶

Urgent and High Priority Summaries

Deadline-Driven Summaries¶

First Response Drafts¶

Evaluation¶

LLM as a Judge¶

Categorization Accuracy¶

Base Categories¶

Critical vs. Non-Critical Accuracy¶

Full Output¶

📧 Executive Summary of Emails Received

Number of Emails from Yesterday: 51

Breakdown of Emails by Category:

Key Entities by Category:

AI Conclusion

Urgent and High Priority Summaries

Summary and Recommendation¶

Summary of Observations¶

Evaluation Highlights¶

Original Email:¶

Summary:¶

Original Email:¶

Summary:¶

Original Email:¶

Summary:¶

Original Email:¶

Summary:¶

Original Email:¶

Summary:¶

Relevance¶

Clarity¶

Actionability¶

Strengths¶

Improvement Areas¶

Final Recommendation¶

Cancer Diagnosis: Malignant vs. Benign Tumors

Mike Anderson

mikeanderson0289@gmail.com