third_party/opencv
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #17502 from dmatveev:dm/infer2

Browse Source 
* G-API: Introduce a new gapi::infer2 overload + gaze estimation sample

* G-API/infer2: Introduced static type checking for infer2

- Also added extra tests on the type check routine

* G-API/infer2: Addressed self-review comments in the sample app

- Also fix build on Linux;

* G-API/infer2: Remove incorrect SetLayout(HWC) + dead code

- Also fixed comments in the backend

* G-API/infer2: Continue with self-review

- Fix warnings/compile errors in gaze estimation
- Dropped the use of RTTI (VectorRef::holds()) from the giebackend
- Replaced it with a trait-based enums for GArray<T> and std::vector<T>
- The enums and traits are temporary and need to be unified with
  the S11N when it comes

* G-API/infer2: Final self-review items

- Refactored ROIList test to cover 70% for infer<> and infer2<>;
- Fixed the model data discovery routine to be compatible with new
  OpenVINO;
- Hopefully fixed the final issues (warnings) with the sample.

* G-API/infer2: address review problems

- Fixed typo in comments;
- Fixed public (Doxygen) comment on GArray<GMat> input case for infer2;
- Made model lookup more flexible to allow new & old OMZ dir layouts.

* G-API/infer2: Change the model paths again

* G-API/infer2: Change the lookup path for test data

* G-API/infer2: use randu instead of imread. CI war is over
This commit is contained in:
Dmitry Matveev
2020-07-14 11:06:49 +03:00
committed by GitHub
parent e5e767abc1
commit f0c411d8b5
14 changed files with 1066 additions and 85 deletions
Download Patch File Download Diff File Expand all files Collapse all files
modules/gapi/include/opencv2/gapi/garray.hpp
+41
View File
@@ -49,6 +49,31 @@ std::ostream& operator<<(std::ostream& os, const cv::GArrayDesc &desc);
namespace detail
{
// FIXME: This type spec needs to be:
// 1) shared with GOpaque (not needed right now)
// 2) unified with the serialization (S11N, not merged right now).
// Adding it to type traits is problematic due to our header deps
// (which also need to be fixed).
enum class TypeSpec: int {
OPAQUE_SPEC,
MAT,
RECT
};
// FIXME: Reuse the below from "opaque traits" of S11N!
template<typename T> struct GTypeSpec;
template<typename T> struct GTypeSpec
{
static constexpr const TypeSpec spec = TypeSpec::OPAQUE_SPEC;
};
template<> struct GTypeSpec<cv::Mat>
{
static constexpr const TypeSpec spec = TypeSpec::MAT;
};
template<> struct GTypeSpec<cv::Rect>
{
static constexpr const TypeSpec spec = TypeSpec::RECT;
};
// ConstructVec is a callback which stores information about T and is used by
// G-API runtime to construct arrays in host memory (T remains opaque for G-API).
// ConstructVec is carried into G-API internals by GArrayU.
@@ -110,12 +135,15 @@ namespace detail
class BasicVectorRef
{
public:
// These fields are set by the derived class(es)
std::size_t m_elemSize = 0ul;
cv::GArrayDesc m_desc;
TypeSpec m_spec;
virtual ~BasicVectorRef() {}
virtual void mov(BasicVectorRef &ref) = 0;
virtual const void* ptr() const = 0;
virtual std::size_t size() const = 0;
};
template<typename T> class VectorRefT final: public BasicVectorRef
@@ -135,6 +163,7 @@ namespace detail
{
m_elemSize = sizeof(T);
if (vec) m_desc = cv::descr_of(*vec);
m_spec = GTypeSpec<T>::spec;
}
public:
@@ -209,7 +238,9 @@ namespace detail
wref() = std::move(tv->wref());
}
virtual const void* ptr() const override { return &rref(); }
virtual std::size_t size() const override { return rref().size(); }
};
// This class strips type information from VectorRefT<> and makes it usable
@@ -265,8 +296,18 @@ namespace detail
return m_ref->m_desc;
}
std::size_t size() const
{
return m_ref->size();
}
// May be used to uniquely identify this object internally
const void *ptr() const { return m_ref->ptr(); }
TypeSpec spec() const
{
return m_ref->m_spec;
}
};
// Helper (FIXME: work-around?)
modules/gapi/include/opencv2/gapi/gkernel.hpp
+16 -3
View File
@@ -26,6 +26,7 @@
namespace cv {
using GSpecs = std::vector<cv::detail::ArgSpec>;
using GShapes = std::vector<GShape>;
// GKernel describes kernel API to the system
@@ -38,8 +39,11 @@ struct GAPI_EXPORTS GKernel
const std::string name; // kernel ID, defined by its API (signature)
const std::string tag; // some (implementation-specific) tag
const M outMeta; // generic adaptor to API::outMeta(...)
const GSpecs inSpecs; // specs of kernel's inputs (FIXME: below)
const GShapes outShapes; // types (shapes) kernel's outputs
};
// TODO: It's questionable if inSpecs should really be here. Instead,
// this information could come from meta.
// GKernelImpl describes particular kernel implementation to the system
struct GAPI_EXPORTS GKernelImpl
@@ -203,10 +207,15 @@ public:
using InArgs = std::tuple<Args...>;
using OutArgs = std::tuple<R...>;
// TODO: Args&&... here?
static std::tuple<R...> on(Args... args)
{
cv::GCall call(GKernel{K::id(), K::tag(), &K::getOutMeta, {detail::GTypeTraits<R>::shape...}});
call.pass(args...);
cv::GCall call(GKernel{ K::id()
, K::tag()
, &K::getOutMeta
, {detail::GTypeTraits<Args>::spec...}
, {detail::GTypeTraits<R>::shape...}});
call.pass(args...); // TODO: std::forward() here?
return yield(call, typename detail::MkSeq<sizeof...(R)>::type());
}
};
@@ -226,7 +235,11 @@ public:
static R on(Args... args)
{
cv::GCall call(GKernel{K::id(), K::tag(), &K::getOutMeta, {detail::GTypeTraits<R>::shape}});
cv::GCall call(GKernel{ K::id()
, K::tag()
, &K::getOutMeta
, {detail::GTypeTraits<Args>::spec...}
, {detail::GTypeTraits<R>::shape}});
call.pass(args...);
return detail::Yield<R>::yield(call, 0);
}
modules/gapi/include/opencv2/gapi/gopaque.hpp
+1
View File
@@ -46,6 +46,7 @@ std::ostream& operator<<(std::ostream& os, const cv::GOpaqueDesc &desc);
namespace detail
{
// ConstructOpaque is a callback which stores information about T and is used by
// G-API runtime to construct an object in host memory (T remains opaque for G-API).
// ConstructOpaque is carried into G-API internals by GOpaqueU.
modules/gapi/include/opencv2/gapi/gtype_traits.hpp
+38 -1
View File
@@ -2,7 +2,7 @@
// It is subject to the license terms in the LICENSE file found in the top-level directory
// of this distribution and at http://opencv.org/license.html.
//
// Copyright (C) 2018 Intel Corporation
// Copyright (C) 2018-2020 Intel Corporation
#ifndef OPENCV_GAPI_GTYPE_TRAITS_HPP
@@ -41,6 +41,32 @@ namespace detail
GOPAQUE, // a cv::GOpaqueU (note - exactly GOpaqueU, not GOpaque<T>!)
};
// This enum captures some information about T in GArray<T> and GOpaque<T>
enum class ArgSpec: int
{
OPAQUE_SPEC, // Unknown, generic, opaque-to-GAPI data type
GMAT, // a GMat
RECT, // a cv::Rect
// NB: Add more types when required
};
// Describe specialization types of interest first
// FIXME: It comes to GArg but ideally it should go to *Desc{}
// type family. Bringing it there is a more massive change though.
template<typename T> struct GSpecTraits;
template<typename T> struct GSpecTraits
{
static constexpr const ArgSpec spec = ArgSpec::OPAQUE_SPEC;
};
template<> struct GSpecTraits<cv::GMat>
{
static constexpr const ArgSpec spec = ArgSpec::GMAT;
};
template<> struct GSpecTraits<cv::Rect>
{
static constexpr const ArgSpec spec = ArgSpec::RECT;
};
enum class OpaqueKind: int
{
CV_UNKNOWN, // Unknown, generic, opaque-to-GAPI data type unsupported in graph seriallization
@@ -69,35 +95,44 @@ namespace detail
// cv::GArg to store meta information about types passed into
// operation arguments. Please note that cv::GComputation is
// defined on GProtoArgs, not GArgs!
//
// spec is a type specialization (makes sense for GArray<> and GOpaque<>)
// for the rest, it is just OPAQUE_VAL by default.
template<typename T> struct GTypeTraits;
template<typename T> struct GTypeTraits
{
static constexpr const ArgKind kind = ArgKind::OPAQUE_VAL;
static constexpr const ArgSpec spec = ArgSpec::OPAQUE_SPEC;
};
template<> struct GTypeTraits<cv::GMat>
{
static constexpr const ArgKind kind = ArgKind::GMAT;
static constexpr const GShape shape = GShape::GMAT;
static constexpr const ArgSpec spec = ArgSpec::OPAQUE_SPEC;
};
template<> struct GTypeTraits<cv::GMatP>
{
static constexpr const ArgKind kind = ArgKind::GMATP;
static constexpr const GShape shape = GShape::GMAT;
static constexpr const ArgSpec spec = ArgSpec::OPAQUE_SPEC;
};
template<> struct GTypeTraits<cv::GFrame>
{
static constexpr const ArgKind kind = ArgKind::GFRAME;
static constexpr const GShape shape = GShape::GMAT;
static constexpr const ArgSpec spec = ArgSpec::OPAQUE_SPEC;
};
template<> struct GTypeTraits<cv::GScalar>
{
static constexpr const ArgKind kind = ArgKind::GSCALAR;
static constexpr const GShape shape = GShape::GSCALAR;
static constexpr const ArgSpec spec = ArgSpec::OPAQUE_SPEC;
};
template<class T> struct GTypeTraits<cv::GArray<T> >
{
static constexpr const ArgKind kind = ArgKind::GARRAY;
static constexpr const GShape shape = GShape::GARRAY;
static constexpr const ArgSpec spec = GSpecTraits<T>::spec;
using host_type = std::vector<T>;
using strip_type = cv::detail::VectorRef;
static cv::detail::GArrayU wrap_value(const cv::GArray<T> &t) { return t.strip();}
@@ -108,6 +143,7 @@ namespace detail
{
static constexpr const ArgKind kind = ArgKind::GOPAQUE;
static constexpr const GShape shape = GShape::GOPAQUE;
static constexpr const ArgSpec spec = GSpecTraits<T>::spec;
using host_type = T;
using strip_type = cv::detail::OpaqueRef;
static cv::detail::GOpaqueU wrap_value(const cv::GOpaque<T> &t) { return t.strip();}
@@ -140,6 +176,7 @@ namespace detail
template<> struct GTypeOf<cv::Scalar> { using type = cv::GScalar; };
template<typename U> struct GTypeOf<std::vector<U> > { using type = cv::GArray<U>; };
template<typename U> struct GTypeOf { using type = cv::GOpaque<U>;};
// FIXME: This is not quite correct since IStreamSource may produce not only Mat but also Scalar
// and vector data. TODO: Extend the type dispatching on these types too.
template<> struct GTypeOf<cv::gapi::wip::IStreamSource::Ptr> { using type = cv::GMat;};
modules/gapi/include/opencv2/gapi/infer.hpp
+108 -2
View File
@@ -14,6 +14,7 @@
#include <functional>
#include <string> // string
#include <utility> // tuple
#include <type_traits> // is_same, false_type
#include <opencv2/gapi/util/any.hpp> // any<>
#include <opencv2/gapi/gkernel.hpp> // GKernelType[M], GBackend
@@ -25,6 +26,43 @@ namespace cv {
template<typename, typename> class GNetworkType;
namespace detail {
template<typename, typename>
struct valid_infer2_types;
// Terminal case 1 (50/50 success)
template<typename T>
struct valid_infer2_types< std::tuple<cv::GMat>, std::tuple<T> > {
// By default, Nets are limited to GMat argument types only
// for infer2, every GMat argument may translate to either
// GArray<GMat> or GArray<Rect>. GArray<> part is stripped
// already at this point.
static constexpr const auto value =
std::is_same<typename std::decay<T>::type, cv::GMat>::value
|| std::is_same<typename std::decay<T>::type, cv::Rect>::value;
};
// Terminal case 2 (100% failure)
template<typename... Ts>
struct valid_infer2_types< std::tuple<>, std::tuple<Ts...> >
: public std::false_type {
};
// Terminal case 3 (100% failure)
template<typename... Ns>
struct valid_infer2_types< std::tuple<Ns...>, std::tuple<> >
: public std::false_type {
};
// Recursion -- generic
template<typename... Ns, typename T, typename...Ts>
struct valid_infer2_types< std::tuple<cv::GMat,Ns...>, std::tuple<T,Ts...> > {
static constexpr const auto value =
valid_infer2_types< std::tuple<cv::GMat>, std::tuple<T> >::value
&& valid_infer2_types< std::tuple<Ns...>, std::tuple<Ts...> >::value;
};
} // namespace detail
// TODO: maybe tuple_wrap_helper from util.hpp may help with this.
// Multiple-return-value network definition (specialized base class)
template<typename K, typename... R, typename... Args>
@@ -39,6 +77,15 @@ public:
using ResultL = std::tuple< cv::GArray<R>... >;
using APIList = std::function<ResultL(cv::GArray<cv::Rect>, Args...)>;
// APIList2 is also template to allow different calling options
// (GArray<cv::Rect> vs GArray<cv::GMat> per input)
template<class... Ts>
using APIList2 = typename std::enable_if
< cv::detail::valid_infer2_types< std::tuple<Args...>
, std::tuple<Ts...> >::value,
std::function<ResultL(cv::GMat, cv::GArray<Ts>...)>
>::type;
};
// Single-return-value network definition (specialized base class)
@@ -54,6 +101,15 @@ public:
using ResultL = cv::GArray<R>;
using APIList = std::function<ResultL(cv::GArray<cv::Rect>, Args...)>;
// APIList2 is also template to allow different calling options
// (GArray<cv::Rect> vs GArray<cv::GMat> per input)
template<class... Ts>
using APIList2 = typename std::enable_if
< cv::detail::valid_infer2_types< std::tuple<Args...>
, std::tuple<Ts...> >::value,
std::function<ResultL(cv::GMat, cv::GArray<Ts>...)>
>::type;
};
// Base "Infer" kernel. Note - for whatever network, kernel ID
@@ -77,10 +133,21 @@ struct GInferBase {
// All notes from "Infer" kernel apply here as well.
struct GInferListBase {
static constexpr const char * id() {
return "org.opencv.dnn.infer-roi"; // Universal stub
return "org.opencv.dnn.infer-roi"; // Universal stub
}
static GMetaArgs getOutMeta(const GMetaArgs &, const GArgs &) {
return GMetaArgs{}; // One more universal stub
return GMetaArgs{}; // One more universal stub
}
};
// Base "Infer list 2" kernel.
// All notes from "Infer" kernel apply here as well.
struct GInferList2Base {
static constexpr const char * id() {
return "org.opencv.dnn.infer-roi-list"; // Universal stub
}
static GMetaArgs getOutMeta(const GMetaArgs &, const GArgs &) {
return GMetaArgs{}; // One more universal stub
}
};
@@ -109,6 +176,21 @@ struct GInferList final
static constexpr const char* tag() { return Net::tag(); }
};
// An even more generic roi-list inference kernel. API (::on()) is
// derived from the Net template parameter (see more in infer<>
// overload).
// Takes an extra variadic template list to reflect how this network
// was called (with Rects or GMats as array parameters)
template<typename Net, typename... Args>
struct GInferList2 final
: public GInferList2Base
, public detail::KernelTypeMedium< GInferList2<Net, Args...>
, typename Net::template APIList2<Args...> > {
using GInferList2Base::getOutMeta; // FIXME: name lookup conflict workaround?
static constexpr const char* tag() { return Net::tag(); }
};
} // namespace cv
// FIXME: Probably the <API> signature makes a function/tuple/function round-trip
@@ -139,6 +221,30 @@ typename Net::ResultL infer(cv::GArray<cv::Rect> roi, Args&&... args) {
return GInferList<Net>::on(roi, std::forward<Args>(args)...);
}
/** @brief Calculates responses for the specified network (template
* parameter) for every region in the source image, extended version.
*
* @tparam A network type defined with G_API_NET() macro.
* @param image A source image containing regions of interest
* @param args GArray<> objects of cv::Rect or cv::GMat, one per every
* network input:
* - If a cv::GArray<cv::Rect> is passed, the appropriate
* regions are taken from `image` and preprocessed to this particular
* network input;
* - If a cv::GArray<cv::GMat> is passed, the underlying data traited
* as tensor (no automatic preprocessing happen).
* @return a list of objects of return type as defined in G_API_NET().
* If a network has multiple return values (defined with a tuple), a tuple of
* GArray<> objects is returned with the appropriate types inside.
* @sa G_API_NET()
*/
template<typename Net, typename... Args>
typename Net::ResultL infer2(cv::GMat image, cv::GArray<Args>... args) {
// FIXME: Declared as "2" because in the current form it steals
// overloads from the regular infer
return GInferList2<Net, Args...>::on(image, args...);
}
/**
* @brief Calculates response for the specified network (template
* parameter) given the input data.
modules/gapi/include/opencv2/gapi/infer/ie.hpp
+2 -2
View File
@@ -78,8 +78,8 @@ public:
const std::string &weights,
const std::string &device)
: desc{ model, weights, device, {}, {}, {}
, std::tuple_size<typename Net::InArgs>::value
, std::tuple_size<typename Net::OutArgs>::value
, std::tuple_size<typename Net::InArgs>::value // num_in
, std::tuple_size<typename Net::OutArgs>::value // num_out
} {
};