In Human-Robot Collaboration, robots are expected to work next to human in warehouses, daily housekeeping, and other robot assistant applications safely, intelligently and friendly. To achieve this goal, the robotic system should be equipped with capacities of understanding intentions of human partners and reasoning according to the behaviors of human partners and the state of the environment. The main idea is to combine the learning-based approach with traditional high-level task planing algorithms. The first step is to build a human model using data collected from visual perception system such as stereo cameras. Based on the learned human model, robots could infer intentions of human partners using the data collected during run-time.