About the role

Aplyr's Quick Take

This role is for a seasoned Python Automation Engineer focused on building and managing web scraping solutions using AWS. You'll be automating complex interactions with secure websites and architecting scalable scraping pipelines. It's an individual contributor role that requires deep technical expertise in Python and cloud services.

Good fit

Ideal candidates have over 10 years of experience in Python development, particularly in automation and data extraction. A strong background in web scraping and cloud deployment, along with a hands-on, problem-solving approach, will help you thrive here.

Worth noting

The position emphasizes advanced web scraping techniques and requires handling complex authentication flows, which may be challenging for those without specific experience in this area. The salary range is competitive for the level of expertise required.

About the Role

iCapital is looking to hire an experienced Python Developer with expertise in web scraping, automated portal interactions, and cloud-native deployment using AWS. The ideal candidate will have hands-on experience working with Playwright for browser automation, managing multi-factor authentication (MFA) flows, and deploying scalable scraping tasks via AWS Lambda and related services. This role will assist in architecting and building robust, secure, and scalable scraping solutions that interact with complex web applications and secured portals.

Responsibilities

Assist with the design and implementation of advanced scraping solutions using Python, Playwright, and AWS services.
Automate interactions with JavaScript-heavy and authentication-secured websites, including handling MFA, CAPTCHAs, and session or token-based login flows.
Architect scraping pipelines using serverless AWS components such as Lambda, Step Functions, S3, CloudWatch, and Secrets Manager.
Build systems that scale to support high volumes of data extraction with fault tolerance, retries, and intelligent logging.
Integrate and manage complex workflows across multiple portals, APIs, and data sources.
Contribute to architectural decisions, tooling, and best practices.

Qualifications

Bachelor’s or master’s degree in computer science, software engineering, or related field, or equivalent professional experience
10+ years of experience in Python development, with a strong focus on automation and data extraction
Proven expertise in web scraping using tools like Playwright, Selenium, Scrapy, BeautifulSoup, and requests
In-depth experience handling multi-step authentication flows, including multi-factor authentication (MFA), CAPTCHA solving, and session or cookie management
Proficient in deploying and managing scraping workloads in AWS, particularly Lambda, S3, IAM, CloudWatch, and Secrets Manager
Experience with asynchronous programming, headless browsers, and JavaScript-rendered content
Solid understanding of web protocols (i.e. HTTP, HTTPS, cookies, headers), and the ability to reverse-engineer network calls and authentication mechanisms
Comfortable working with APIs, JSON/XML, and data transformation
Experience with CI or CD pipelines, Docker, and infrastructure as code (i.e. CloudFormation, Terraform)
Familiar with data pipeline orchestration tools such as Apache Airflow or AWS Step Functions
Prior experience scraping from secure or enterprise-level portals (i.e. financial, healthcare, legal)
Background in data engineering or ETL workflows is a plus
Exposure to Python testing frameworks and writing unit or integration tests

Benefits

The base salary range for this role is $130,000 to $160,000. iCapital offers a compensation package which includes salary, equity for all full-time employees, and an annual performance bonus. Employees also receive a comprehensive benefits package that includes an employer matched retirement plan, generously subsidized healthcare with 100% employer paid dental, vision, telemedicine, and virtual mental health counseling, parental leave, and unlimited paid time off (PTO).

We believe the best ideas and innovation happen when we are together. Employees in this role will work in the office Monday-Thursday, with the flexibility to work remotely on Friday.

For additional information on iCapital, please visit https://www.icapitalnetwork.com/about-us Twitter: @icapitalnetwork | LinkedIn: https://www.linkedin.com/company/icapital-network-inc | Awards Disclaimer: https://www.icapitalnetwork.com/about-us/recognition/

iCapital is proud to be an Equal Employment Opportunity and Affirmative Action employer. We do not discriminate based upon race, religion, color, national origin, gender, sexual orientation, gender identity, age, status as a protected veteran, status as an individual with a disability, or other applicable legally protected characteristics.

Skills & Tags

python java javascript go aws docker ai data

Aplyr's read

iCapital revolutionizes access to alternative investments, attracting professionals skilled in financial technology and relationship management.
Synthesized from recent postings & public sources

What's promising

•iCapital's platform simplifies access to complex alternative investments for wealth managers.
•The company is expanding rapidly, hiring for multiple senior roles across various departments.
•iCapital's focus on fintech innovation positions it well in the evolving financial services landscape.

What to watch

•The niche focus on alternative investments may limit broader market appeal.
•High reliance on senior roles suggests potential challenges in junior talent development.
•Rapid expansion could strain resources and affect operational efficiency.

Why iCapital

•iCapital offers a specialized platform that democratizes alternative investment access.
•The company's strong emphasis on fintech solutions differentiates it from traditional financial service firms.
•iCapital's targeted regional relationship management strategy enhances client engagement.

Aplyr’s read is generated by AI from public sources. Was it useful?