LLM Systems Engineer - AI Dept.
About the position PLEXIS Healthcare Systems, Inc., a progressive software development company in Medford, OR, is looking to hire an enthusiastic and experienced LLM Systems Engineer within our AI Department. The candidate must be local to the Southern Oregon/Rogue Valley area. Please apply at www.plexishealth.com Position Purpose The LLM Systems Engineer will focus on the design, implementation, and operation of applied Large Language Model (LLM) systems used in production software. The position centers on how LLMs retrieve information, manage context and memory, apply behavioral guardrails, and operate safely with appropriate human oversight. This role does not focus on training or fine-tuning foundation models, but instead on building reliable, observable, and compliant systems around them. The AI Special Ops team is a dynamic and experimental group focused on exploring cutting-edge AI capabilities and translating them into real-world, production-ready solutions. The LLM Systems Engineer will collaborate closely with engineers, product leaders, and end users to help ensure AI systems are accurate, safe, and trustworthy. Responsibilities • Design, build, and maintain Retrieval-Augmented Generation (RAG) pipelines for LLM-powered features • Ingest, structure, and manage data from PDFs, document repositories, databases, and semi-structured sources • Implement chunking strategies, embeddings, vector search, and re-ranking to improve retrieval precision • Develop and maintain context and memory mechanisms for LLM interactions, including relevance and summarization strategies • Apply behavioral and domain guardrails to guide model outputs and prevent inappropriate or unsafe responses • Support Human-in-the-Loop (HITL) escalation workflows for sensitive or ambiguous scenarios • Capture and incorporate explicit and implicit user feedback to improve system behavior over time • Implement logging and observability for LLM inputs, retrieved context, outputs, and guardrail events • Assist with debugging, incident review, and performance tuning of AI systems • Collaborate with cross-functional team members in an agile development environment • Duties are subject to change as projects and priorities evolve • Required to work a minimum of 3 days per week in the office; remaining days may be worked remotely or in-office Requirements • 13 years of professional software development experience • Experience as a backend or full-stack engineer working with APIs and data pipelines • Hands-on exposure to LLMs and applied AI systems through professional work, side projects, or prototypes • Working understanding of embeddings, vector search, and LLM context window limitations • Familiarity with Retrieval-Augmented Generation (RAG) concepts • Ability to reason about model accuracy, hallucinations, and system limitations • Strong problem-solving skills and comfort working in ambiguous, exploratory problem spaces • Effective communication skills and ability to collaborate across technical and non-technical teams Nice-to-haves • Experience with LLM tooling or frameworks (e.g., LangChain, LlamaIndex, or similar) • Exposure to regulated or high-trust domains such as healthcare, insurance, or finance • Familiarity with logging, monitoring, or observability tooling • Experience contributing to production systems involving AI or automation Benefits • Competitive Healthcare Package; Medical, Dental, Vision and 401K • Competitive Vacation and Sick Time Package Apply tot his job